2024-09-21T00:38:22.782 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T00:38:22.809 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T00:38:22.906 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344 branch: wip-pdonnell-testing-20240920.212106-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7913344' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240920.212106-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy mgr-modules: - snap_schedule sha1: 1843221895adccfb3b692060251dd6a0183bc410 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 1843221895adccfb3b692060251dd6a0183bc410 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240920.212106-debug sha1: 1843221895adccfb3b692060251dd6a0183bc410 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7864 sha1: 1843221895adccfb3b692060251dd6a0183bc410 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240920.212106-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1843221895adccfb3b692060251dd6a0183bc410 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNm/KcTyAE5D6icLRUGlM80gfBt8FznzoBmJDWLYIdYSgsl8DMiW+r9I+MFTibn2HPLzJr+zYClkXkPkYGqynFg= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE+noCh/0/7zPuopIZLQl/anubNVTfGBAIJ4QplWFj1AlqbHdDNxAWpnbNJkmPtg30CgMvaXmhmD4qMvqZzB6wQ= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJyWdKch69yeJ47h0frg6AkdKIShT0OQlH2b4bm7i9BZktJ5h/PyDew2FlASrsaBuKzvupAAfhMH2EwGPh9YKw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_23:48:22 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T00:38:22.906 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa; will attempt to use it 2024-09-21T00:38:22.907 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks 2024-09-21T00:38:22.907 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T00:38:22.909 INFO:teuthology.task.internal:Checking packages... 2024-09-21T00:38:22.929 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1843221895adccfb3b692060251dd6a0183bc410' 2024-09-21T00:38:22.929 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T00:38:22.929 INFO:teuthology.packaging:ref: None 2024-09-21T00:38:22.929 INFO:teuthology.packaging:tag: None 2024-09-21T00:38:22.929 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240920.212106-debug 2024-09-21T00:38:22.929 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:38:22.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240920.212106-debug 2024-09-21T00:38:23.072 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5120.g18432218 2024-09-21T00:38:23.076 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T00:38:23.082 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T00:38:23.082 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T00:38:23.110 INFO:teuthology.task.internal:Saving configuration 2024-09-21T00:38:23.130 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T00:38:23.136 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T00:38:23.160 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 00:32:29.824600', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNm/KcTyAE5D6icLRUGlM80gfBt8FznzoBmJDWLYIdYSgsl8DMiW+r9I+MFTibn2HPLzJr+zYClkXkPkYGqynFg='} 2024-09-21T00:38:23.184 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 00:32:29.826276', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE+noCh/0/7zPuopIZLQl/anubNVTfGBAIJ4QplWFj1AlqbHdDNxAWpnbNJkmPtg30CgMvaXmhmD4qMvqZzB6wQ='} 2024-09-21T00:38:23.209 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 00:32:29.822691', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJyWdKch69yeJ47h0frg6AkdKIShT0OQlH2b4bm7i9BZktJ5h/PyDew2FlASrsaBuKzvupAAfhMH2EwGPh9YKw='} 2024-09-21T00:38:23.209 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T00:38:23.215 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-21T00:38:23.215 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-21T00:38:23.215 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-21T00:38:23.215 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T00:38:23.321 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd6b2ee4160>, signals=[15]) 2024-09-21T00:38:23.321 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T00:38:23.326 INFO:teuthology.task.internal:Opening connections... 2024-09-21T00:38:23.326 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T00:38:23.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:38:23.406 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-09-21T00:38:23.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:38:23.482 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T00:38:23.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:38:23.557 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T00:38:23.562 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-21T00:38:23.580 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-21T00:38:23.580 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-09-21T00:38:23.638 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T00:38:23.639 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T00:38:23.639 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T00:38:23.639 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-21T00:38:23.666 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-09-21T00:38:23.684 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-09-21T00:38:23.685 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T00:38:23.743 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-09-21T00:38:23.744 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-09-21T00:38:23.744 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T00:38:23.744 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-09-21T00:38:23.744 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T00:38:23.744 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T00:38:23.744 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T00:38:23.744 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-09-21T00:38:23.765 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-09-21T00:38:23.784 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-09-21T00:38:23.784 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-09-21T00:38:23.842 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T00:38:23.843 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T00:38:23.843 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T00:38:23.843 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-09-21T00:38:23.866 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T00:38:23.876 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T00:38:23.901 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T00:38:23.902 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-21T00:38:23.904 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-09-21T00:38:23.906 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-09-21T00:38:23.923 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T00:38:23.929 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T00:38:23.929 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-21T00:38:23.961 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-09-21T00:38:23.964 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-09-21T00:38:24.073 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T00:38:24.140 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T00:38:24.185 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-21T00:38:24.185 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-09-21T00:38:24.185 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-09-21T00:38:24.185 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T00:38:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:38:24.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-21T00:38:24.363 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-21T00:38:24.363 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-21T00:38:24.363 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-21T00:38:24.380 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-503.el9.x86_64 2024-09-21T00:38:24.381 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-21T00:38:24.381 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-21T00:38:24.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum upgrade -y linux-firmware 2024-09-21T00:38:28.144 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-21T00:38:33.227 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 20 MB 00:03 2024-09-21T00:38:37.753 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-21T00:38:39.970 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 19 kB 00:00 2024-09-21T00:38:40.791 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-21T00:38:46.397 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T00:38:48.470 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-21T00:38:48.475 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-21T00:38:48.475 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware noarch 20240905-146.el9 baseos 388 M 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 1 Package 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 388 M 2024-09-21T00:38:48.476 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-21T00:38:54.507 INFO:teuthology.orchestra.run.smithi039.stdout:linux-firmware-20240905-146.el9.noarch.rpm 65 MB/s | 388 MB 00:06 2024-09-21T00:38:54.507 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-21T00:38:54.507 INFO:teuthology.orchestra.run.smithi039.stdout:Total 64 MB/s | 388 MB 00:06 2024-09-21T00:38:57.778 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-21T00:38:57.880 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-21T00:38:57.880 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-21T00:39:01.310 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-21T00:39:01.310 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-21T00:39:11.380 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-21T00:39:11.744 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-21T00:39:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-21T00:39:12.661 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: linux-firmware-20240716-145.el9.noarch 2/2 2024-09-21T00:39:12.661 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-21T00:39:13.503 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-21T00:39:13.503 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:39:13.503 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-09-21T00:39:13.503 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware-20240905-146.el9.noarch 2024-09-21T00:39:13.503 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:39:13.503 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-21T00:39:13.843 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-21T00:39:13.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-21T00:39:13.989 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-21T00:39:13.989 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-21T00:39:13.992 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi039.front.sepia.ceph.com to complete... 2024-09-21T00:39:20.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21 00:39:20 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-21T00:39:20.297 INFO:teuthology.orchestra.run.smithi039.stderr:FINISHED --2024-09-21 00:39:20-- 2024-09-21T00:39:20.297 INFO:teuthology.orchestra.run.smithi039.stderr:Total wall clock time: 6.3s 2024-09-21T00:39:20.297 INFO:teuthology.orchestra.run.smithi039.stderr:Downloaded: 1 files, 343M in 6.0s (57.2 MB/s) 2024-09-21T00:39:20.300 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-21T00:39:20.300 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-21T00:39:22.461 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying... ######################################## 2024-09-21T00:39:22.526 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing... ######################################## 2024-09-21T00:39:22.589 INFO:teuthology.orchestra.run.smithi039.stdout:Updating / installing... 2024-09-21T00:39:30.912 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-21T00:40:11.451 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-21T00:40:11.451 DEBUG:teuthology.orchestra.run.smithi039:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-21T00:40:11.539 INFO:teuthology.orchestra.run.smithi039.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-21T00:40:11.540 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-21T00:40:11.541 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-21T00:40:11.542 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-21T00:40:11.543 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-21T00:40:11.544 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-21T00:40:11.545 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-21T00:40:11.546 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-21T00:40:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-21T00:40:11.548 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-21T00:40:11.549 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-21T00:40:11.550 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-21T00:40:11.551 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-21T00:40:11.552 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-21T00:40:11.553 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-21T00:40:11.554 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-21T00:40:11.555 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-21T00:40:11.556 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-21T00:40:11.557 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-21T00:40:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-21T00:40:11.559 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-21T00:40:11.560 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-21T00:40:11.561 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-21T00:40:11.562 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-21T00:40:11.563 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-21T00:40:11.564 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-21T00:40:11.565 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-21T00:40:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-21T00:40:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-21T00:40:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-21T00:40:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-21T00:40:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-21T00:40:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-21T00:40:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-21T00:40:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-21T00:40:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-21T00:40:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-21T00:40:11.576 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-21T00:40:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-21T00:40:11.578 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-21T00:40:11.579 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-21T00:40:11.580 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-21T00:40:11.581 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-21T00:40:11.582 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-21T00:40:11.583 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-21T00:40:11.584 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-21T00:40:11.585 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-21T00:40:11.586 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-21T00:40:11.587 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-21T00:40:11.588 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-21T00:40:11.589 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-21T00:40:11.590 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-21T00:40:11.591 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-21T00:40:11.592 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-21T00:40:11.593 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-21T00:40:11.594 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-21T00:40:11.595 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-21T00:40:11.596 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-21T00:40:11.597 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-21T00:40:11.598 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-21T00:40:11.599 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-21T00:40:11.600 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-21T00:40:11.601 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-21T00:40:11.602 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-21T00:40:11.603 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-21T00:40:11.604 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-21T00:40:11.605 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-21T00:40:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-21T00:40:11.607 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-21T00:40:11.608 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-21T00:40:11.609 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-21T00:40:11.610 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-21T00:40:11.611 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-21T00:40:11.612 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-21T00:40:11.613 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-21T00:40:11.614 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-21T00:40:11.615 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-21T00:40:11.616 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-21T00:40:11.617 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-21T00:40:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-21T00:40:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-21T00:40:11.620 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-21T00:40:11.621 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-21T00:40:11.622 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-21T00:40:11.623 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-21T00:40:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-21T00:40:11.625 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-21T00:40:11.626 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-21T00:40:11.627 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-21T00:40:11.628 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-21T00:40:11.629 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-21T00:40:11.630 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-21T00:40:11.631 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-21T00:40:11.632 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-21T00:40:11.633 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-21T00:40:11.634 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-21T00:40:11.635 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-21T00:40:11.636 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-21T00:40:11.637 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-21T00:40:11.638 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-21T00:40:11.639 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-21T00:40:11.640 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-21T00:40:11.641 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-21T00:40:11.642 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-21T00:40:11.643 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-21T00:40:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-21T00:40:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-21T00:40:11.646 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-21T00:40:11.647 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-21T00:40:11.648 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-21T00:40:11.649 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-21T00:40:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-21T00:40:11.651 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-21T00:40:11.652 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-21T00:40:11.653 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-21T00:40:11.654 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-21T00:40:11.655 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-21T00:40:11.656 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-21T00:40:11.657 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-21T00:40:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-21T00:40:11.659 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-21T00:40:11.660 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-21T00:40:11.661 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-21T00:40:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-21T00:40:11.663 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-21T00:40:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-21T00:40:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-21T00:40:11.666 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-21T00:40:11.667 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-21T00:40:11.668 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-21T00:40:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-21T00:40:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-21T00:40:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-21T00:40:11.672 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-21T00:40:11.673 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-21T00:40:11.674 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-21T00:40:11.675 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-21T00:40:11.676 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-21T00:40:11.677 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-21T00:40:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-21T00:40:11.679 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-21T00:40:11.680 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-21T00:40:11.681 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-21T00:40:11.685 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.685 DEBUG:teuthology.orchestra.run.smithi039:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout:postinstall scriptlet (using /bin/sh): 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout:for file in vmlinuz System.map config; do 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout: fi 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-21T00:40:11.709 INFO:teuthology.orchestra.run.smithi039.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.710 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-09-21T00:40:11.711 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-09-21T00:40:11.785 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-09-21T00:40:11.785 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-09-21T00:40:11.785 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-09-21T00:40:11.785 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 82.el9 2024-09-21T00:40:11.785 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-09-21T00:40:11.785 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8274670 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:40:11.786 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T00:40:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T00:40:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T00:40:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-09-21T00:40:11.787 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:40:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T00:40:11.788 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T00:40:11.788 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:11.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T00:40:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-21T00:40:13.779 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T00:40:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-21T00:40:13.800 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-21T00:40:13.830 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.sl5366DuC2 2024-09-21T00:40:13.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sl5366DuC2 2024-09-21T00:40:13.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.sl5366DuC2 2024-09-21T00:40:14.049 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.sl5366DuC2 is 6KB 2024-09-21T00:40:14.097 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.sl5366DuC2 2024-09-21T00:40:14.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T00:40:14.174 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T00:40:14.174 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T00:40:14.174 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T00:40:14.175 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-21T00:40:14.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:40:14.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-21T00:40:44.320 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-21T00:40:44.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:40:44.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:41:18.831 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T00:41:27.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:41:27.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:41:30.899 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T00:41:42.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:41:42.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:42:42.902 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T00:42:57.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:42:57.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:42:58.282 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-21T00:42:58.812 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:42:58.813 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-21T00:42:58.813 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-21T00:42:58.847 INFO:teuthology.orchestra.run.smithi039.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:42:58.847 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-21T00:42:58.847 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T00:42:58.847 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T00:42:59.849 DEBUG:teuthology.parallel:result is None 2024-09-21T00:42:59.849 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T00:42:59.856 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T00:42:59.856 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T00:42:59.861 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T00:42:59.864 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T00:42:59.895 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T00:42:59.902 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T00:42:59.926 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T00:42:59.926 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T00:42:59.939 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T00:42:59.941 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T00:43:00.053 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T00:43:00.060 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T00:43:00.060 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T00:43:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:43:00.077 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T00:43:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:43:00.095 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T00:43:00.099 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T00:43:00.120 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T00:43:00.158 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T00:43:00.173 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T00:43:00.188 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T00:43:00.202 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T00:43:00.204 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T00:43:00.234 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T00:43:00.238 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T00:43:00.245 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T00:43:00.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T00:43:00.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T00:43:00.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T00:43:00.303 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T00:43:00.313 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T00:43:00.313 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T00:43:00.317 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T00:43:00.320 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T00:43:00.348 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T00:43:00.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T00:43:00.535 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T00:43:00.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T00:43:00.668 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:43:00.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T00:43:00.722 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T00:43:00.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T00:43:00.816 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T00:43:00.900 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T00:43:00.943 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:43:00.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T00:43:01.010 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T00:43:01.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T00:43:01.107 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T00:43:01.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T00:43:01.235 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:43:01.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T00:43:01.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-21T00:43:01.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-09-21T00:43:01.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-09-21T00:43:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T00:43:01.383 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T00:43:01.384 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T00:43:01.891 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T00:43:01.899 INFO:teuthology.task.internal:Starting timer... 2024-09-21T00:43:01.899 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T00:43:01.947 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T00:43:01.979 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T00:43:01.979 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-09-21T00:43:02.068 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-09-21T00:43:02.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-09-21T00:43:02.113 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-09-21T00:43:02.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-09-21T00:43:02.170 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-09-21T00:43:02.172 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T00:43:02.172 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-09-21T00:43:02.235 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-09-21T00:43:02.235 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-09-21T00:43:02.258 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-09-21T00:43:02.259 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-09-21T00:43:02.283 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-09-21T00:43:02.283 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2024-09-21T00:43:02.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T00:43:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:43:02.341 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T00:43:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:43:02.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T00:43:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:43:02.400 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T00:43:02.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-09-21T00:43:02.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-09-21T00:43:02.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-09-21T00:43:02.536 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T00:43:02.546 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T00:43:02.549 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T00:43:02.692 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T00:43:02.702 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T00:43:02.706 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi150.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T00:47:46.807 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-09-21T00:47:46.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:47:46.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:47:46.894 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-21T00:47:47.001 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T00:47:47.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-21T00:47:47.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:47:47.076 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-21T00:47:47.111 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-21T00:47:47.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T00:47:47.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T00:47:47.187 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-21T00:47:47.266 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T00:47:47.267 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T00:47:47.276 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T00:47:47.276 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T00:47:47.277 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T00:47:47.281 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T00:47:47.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T00:47:47.283 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T00:47:47.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T00:47:47.311 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T00:47:47.326 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T00:47:47.337 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T00:47:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T00:47:47.352 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T00:47:47.354 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-09-21T00:47:47.366 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-09-21T00:47:47.369 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T00:47:47.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-09-21T00:47:47.381 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T00:47:47.387 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-09-21T00:47:47.395 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T00:47:47.402 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T00:47:47.416 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T00:47:47.420 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-09-21T00:47:47.444 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-09-21T00:47:47.456 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-09-21T00:47:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T00:47:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-09-21T00:47:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.473 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T00:47:47.474 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-21T00:47:47.477 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T00:47:47.477 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-21T00:47:47.477 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.477 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.477 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.477 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.503 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T00:47:47.596 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-09-21T00:47:47.603 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T00:47:47.604 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-09-21T00:47:47.604 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.604 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.604 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.604 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:47:47.605 INFO:teuthology.run_tasks:Running task install... 2024-09-21T00:47:47.613 DEBUG:teuthology.task.install:project ceph 2024-09-21T00:47:47.614 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}} 2024-09-21T00:47:47.614 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T00:47:47.614 INFO:teuthology.task.install:Using flavor: default 2024-09-21T00:47:47.619 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T00:47:47.619 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-21T00:47:47.620 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T00:47:47.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:47:47.622 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T00:47:47.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:47:47.623 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T00:47:47.623 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:47:47.771 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T00:47:47.771 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T00:47:47.774 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T00:47:47.774 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T00:47:47.788 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T00:47:47.788 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T00:47:47.881 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T00:47:47.881 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:47:47.881 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T00:47:47.890 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T00:47:47.890 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:47:47.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T00:47:47.897 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T00:47:47.897 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:47:47.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T00:47:47.916 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T00:47:47.916 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T00:47:47.924 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T00:47:47.924 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T00:47:47.939 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T00:47:47.939 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T00:47:47.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T00:47:47.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T00:47:48.037 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T00:47:48.059 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T00:47:48.063 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T00:47:48.115 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T00:47:48.123 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2024-09-21T00:47:48.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2024-09-21T00:47:48.127 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-09-21T00:47:48.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-09-21T00:47:48.172 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-09-21T00:47:48.175 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-09-21T00:47:48.350 INFO:teuthology.orchestra.run.smithi160.stdout:45 files removed 2024-09-21T00:47:48.355 INFO:teuthology.orchestra.run.smithi150.stdout:45 files removed 2024-09-21T00:47:48.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T00:47:48.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T00:47:48.465 INFO:teuthology.orchestra.run.smithi039.stdout:45 files removed 2024-09-21T00:47:48.501 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T00:47:48.922 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 320 kB/s | 84 kB 00:00 2024-09-21T00:47:48.924 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 311 kB/s | 84 kB 00:00 2024-09-21T00:47:49.201 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 289 kB/s | 84 kB 00:00 2024-09-21T00:47:49.208 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-21T00:47:49.214 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-21T00:47:49.484 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-21T00:47:49.490 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-21T00:47:49.510 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 92 kB/s | 24 kB 00:00 2024-09-21T00:47:49.793 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-21T00:47:50.192 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-21T00:47:51.182 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.3 MB 00:01 2024-09-21T00:47:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.3 MB 00:01 2024-09-21T00:47:52.927 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-21T00:47:53.386 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 20 MB 00:02 2024-09-21T00:47:53.569 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-21T00:47:57.752 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-21T00:47:58.487 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.5 MB 00:00 2024-09-21T00:47:59.311 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-21T00:47:59.475 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-09-21T00:48:00.229 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 19 kB 00:00 2024-09-21T00:48:00.827 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-21T00:48:01.209 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-21T00:48:01.217 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-21T00:48:02.558 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-21T00:48:06.288 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T00:48:07.724 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T00:48:08.095 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T00:48:08.449 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T00:48:08.449 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T00:48:08.461 INFO:teuthology.orchestra.run.smithi160.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T00:48:08.462 INFO:teuthology.orchestra.run.smithi160.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T00:48:08.462 INFO:teuthology.orchestra.run.smithi160.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T00:48:08.463 INFO:teuthology.orchestra.run.smithi160.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T00:48:08.464 INFO:teuthology.orchestra.run.smithi160.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T00:48:08.467 INFO:teuthology.orchestra.run.smithi160.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T00:48:08.467 INFO:teuthology.orchestra.run.smithi160.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T00:48:08.536 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T00:48:08.550 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-09-21T00:48:08.550 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T00:48:08.550 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-09-21T00:48:08.550 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T00:48:08.550 INFO:teuthology.orchestra.run.smithi160.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T00:48:08.551 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T00:48:08.552 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T00:48:08.553 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T00:48:08.554 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T00:48:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T00:48:08.556 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T00:48:08.557 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T00:48:08.558 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T00:48:08.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T00:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T00:48:08.561 INFO:teuthology.orchestra.run.smithi160.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:Install 149 Packages 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 8 Packages 2024-09-21T00:48:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:08.563 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 228 M 2024-09-21T00:48:08.563 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T00:48:09.173 INFO:teuthology.orchestra.run.smithi160.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 49 kB/s | 6.6 kB 00:00 2024-09-21T00:48:09.350 INFO:teuthology.orchestra.run.smithi160.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 4.9 MB/s | 873 kB 00:00 2024-09-21T00:48:09.407 INFO:teuthology.orchestra.run.smithi160.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 2.5 MB/s | 148 kB 00:00 2024-09-21T00:48:09.540 INFO:teuthology.orchestra.run.smithi160.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-21T00:48:09.599 INFO:teuthology.orchestra.run.smithi160.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 12 MB/s | 2.3 MB 00:00 2024-09-21T00:48:09.674 INFO:teuthology.orchestra.run.smithi160.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 11 MB/s | 1.5 MB 00:00 2024-09-21T00:48:09.974 INFO:teuthology.orchestra.run.smithi160.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 13 MB/s | 5.0 MB 00:00 2024-09-21T00:48:10.183 INFO:teuthology.orchestra.run.smithi160.stdout:(8/157): ceph-common-19.3.0-5120.g18432218.el9. 17 MB/s | 20 MB 00:01 2024-09-21T00:48:10.196 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T00:48:10.197 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T00:48:10.209 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T00:48:10.210 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T00:48:10.211 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T00:48:10.212 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T00:48:10.214 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T00:48:10.216 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T00:48:10.217 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T00:48:10.221 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T00:48:10.222 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T00:48:10.233 INFO:teuthology.orchestra.run.smithi160.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 508 kB/s | 25 kB 00:00 2024-09-21T00:48:10.233 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T00:48:10.234 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T00:48:10.235 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T00:48:10.236 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T00:48:10.237 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T00:48:10.239 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T00:48:10.240 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T00:48:10.287 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-21T00:48:10.302 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-09-21T00:48:10.304 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-09-21T00:48:10.304 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-21T00:48:10.304 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T00:48:10.305 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T00:48:10.306 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T00:48:10.307 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T00:48:10.308 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T00:48:10.309 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T00:48:10.310 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T00:48:10.311 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T00:48:10.312 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T00:48:10.313 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T00:48:10.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T00:48:10.315 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout:Install 149 Packages 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 8 Packages 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 228 M 2024-09-21T00:48:10.316 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-21T00:48:10.318 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-09-21T00:48:10.318 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-09-21T00:48:10.318 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-09-21T00:48:10.318 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-09-21T00:48:10.318 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T00:48:10.318 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T00:48:10.319 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T00:48:10.320 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T00:48:10.321 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T00:48:10.322 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T00:48:10.323 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T00:48:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T00:48:10.325 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T00:48:10.326 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T00:48:10.327 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T00:48:10.328 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-09-21T00:48:10.329 INFO:teuthology.orchestra.run.smithi150.stdout:Install 149 Packages 2024-09-21T00:48:10.330 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 8 Packages 2024-09-21T00:48:10.330 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:48:10.330 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 228 M 2024-09-21T00:48:10.330 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-09-21T00:48:10.650 INFO:teuthology.orchestra.run.smithi160.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 15 MB/s | 10 MB 00:00 2024-09-21T00:48:10.817 INFO:teuthology.orchestra.run.smithi160.stdout:(11/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 16 MB/s | 18 MB 00:01 2024-09-21T00:48:10.842 INFO:teuthology.orchestra.run.smithi160.stdout:(12/157): cephfs-mirror-19.3.0-5120.g18432218.e 1.2 MB/s | 226 kB 00:00 2024-09-21T00:48:10.859 INFO:teuthology.orchestra.run.smithi160.stdout:(13/157): cephfs-shell-19.3.0-5120.g18432218.el 649 kB/s | 27 kB 00:00 2024-09-21T00:48:10.876 INFO:teuthology.orchestra.run.smithi160.stdout:(14/157): libcephfs-devel-19.3.0-5120.g18432218 945 kB/s | 32 kB 00:00 2024-09-21T00:48:10.903 INFO:teuthology.orchestra.run.smithi039.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 43 kB/s | 6.6 kB 00:00 2024-09-21T00:48:10.921 INFO:teuthology.orchestra.run.smithi150.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 55 kB/s | 6.6 kB 00:00 2024-09-21T00:48:10.926 INFO:teuthology.orchestra.run.smithi160.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 3.1 MB/s | 158 kB 00:00 2024-09-21T00:48:10.968 INFO:teuthology.orchestra.run.smithi160.stdout:(16/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.9 MB/s | 766 kB 00:00 2024-09-21T00:48:10.993 INFO:teuthology.orchestra.run.smithi160.stdout:(17/157): librados-devel-19.3.0-5120.g18432218. 1.9 MB/s | 127 kB 00:00 2024-09-21T00:48:11.027 INFO:teuthology.orchestra.run.smithi160.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 8.5 MB/s | 507 kB 00:00 2024-09-21T00:48:11.119 INFO:teuthology.orchestra.run.smithi160.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 493 kB/s | 45 kB 00:00 2024-09-21T00:48:11.219 INFO:teuthology.orchestra.run.smithi160.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 1.5 MB/s | 148 kB 00:00 2024-09-21T00:48:11.238 INFO:teuthology.orchestra.run.smithi150.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 2.7 MB/s | 873 kB 00:00 2024-09-21T00:48:11.330 INFO:teuthology.orchestra.run.smithi150.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 1.6 MB/s | 148 kB 00:00 2024-09-21T00:48:11.336 INFO:teuthology.orchestra.run.smithi160.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 1.4 MB/s | 161 kB 00:00 2024-09-21T00:48:11.428 INFO:teuthology.orchestra.run.smithi160.stdout:(22/157): librgw2-19.3.0-5120.g18432218.el9.x86 12 MB/s | 5.3 MB 00:00 2024-09-21T00:48:11.453 INFO:teuthology.orchestra.run.smithi160.stdout:(23/157): python3-rados-19.3.0-5120.g18432218.e 2.7 MB/s | 321 kB 00:00 2024-09-21T00:48:11.500 INFO:teuthology.orchestra.run.smithi039.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 1.4 MB/s | 873 kB 00:00 2024-09-21T00:48:11.512 INFO:teuthology.orchestra.run.smithi160.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 3.6 MB/s | 307 kB 00:00 2024-09-21T00:48:11.537 INFO:teuthology.orchestra.run.smithi160.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.2 MB/s | 99 kB 00:00 2024-09-21T00:48:11.596 INFO:teuthology.orchestra.run.smithi160.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.0 MB/s | 86 kB 00:00 2024-09-21T00:48:11.650 INFO:teuthology.orchestra.run.smithi039.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 989 kB/s | 148 kB 00:00 2024-09-21T00:48:11.696 INFO:teuthology.orchestra.run.smithi160.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 1.7 MB/s | 170 kB 00:00 2024-09-21T00:48:11.796 INFO:teuthology.orchestra.run.smithi160.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 343 kB/s | 34 kB 00:00 2024-09-21T00:48:11.872 INFO:teuthology.orchestra.run.smithi150.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 5.2 MB/s | 5.5 MB 00:01 2024-09-21T00:48:11.897 INFO:teuthology.orchestra.run.smithi160.stdout:(29/157): rbd-mirror-19.3.0-5120.g18432218.el9. 8.6 MB/s | 3.1 MB 00:00 2024-09-21T00:48:11.922 INFO:teuthology.orchestra.run.smithi160.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 1.2 MB/s | 154 kB 00:00 2024-09-21T00:48:11.939 INFO:teuthology.orchestra.run.smithi150.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 3.8 MB/s | 2.3 MB 00:00 2024-09-21T00:48:12.514 INFO:teuthology.orchestra.run.smithi150.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-09-21T00:48:12.556 INFO:teuthology.orchestra.run.smithi160.stdout:(31/157): ceph-test-19.3.0-5120.g18432218.el9.x 17 MB/s | 40 MB 00:02 2024-09-21T00:48:12.587 INFO:teuthology.orchestra.run.smithi039.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 3.0 MB/s | 5.5 MB 00:01 2024-09-21T00:48:12.614 INFO:teuthology.orchestra.run.smithi160.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 4.9 MB/s | 292 kB 00:00 2024-09-21T00:48:12.634 INFO:teuthology.orchestra.run.smithi039.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 2.3 MB/s | 2.3 MB 00:00 2024-09-21T00:48:12.706 INFO:teuthology.orchestra.run.smithi160.stdout:(33/157): ceph-mgr-diskprediction-local-19.3.0- 9.4 MB/s | 7.4 MB 00:00 2024-09-21T00:48:12.761 INFO:teuthology.orchestra.run.smithi160.stdout:(34/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 338 kB/s | 49 kB 00:00 2024-09-21T00:48:12.857 INFO:teuthology.orchestra.run.smithi160.stdout:(35/157): ceph-prometheus-alerts-19.3.0-5120.g1 110 kB/s | 16 kB 00:00 2024-09-21T00:48:12.890 INFO:teuthology.orchestra.run.smithi160.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 2.2 MB/s | 285 kB 00:00 2024-09-21T00:48:12.940 INFO:teuthology.orchestra.run.smithi039.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 4.3 MB/s | 1.5 MB 00:00 2024-09-21T00:48:12.949 INFO:teuthology.orchestra.run.smithi160.stdout:(37/157): cephadm-19.3.0-5120.g18432218.el9.noa 8.3 MB/s | 784 kB 00:00 2024-09-21T00:48:12.974 INFO:teuthology.orchestra.run.smithi160.stdout:(38/157): cephfs-top-19.3.0-5120.g18432218.el9. 220 kB/s | 18 kB 00:00 2024-09-21T00:48:13.056 INFO:teuthology.orchestra.run.smithi150.stdout:(7/157): ceph-common-19.3.0-5120.g18432218.el9. 8.8 MB/s | 20 MB 00:02 2024-09-21T00:48:13.141 INFO:teuthology.orchestra.run.smithi160.stdout:(39/157): ceph-mgr-dashboard-19.3.0-5120.g18432 11 MB/s | 14 MB 00:01 2024-09-21T00:48:13.193 INFO:teuthology.orchestra.run.smithi039.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 8.9 MB/s | 5.0 MB 00:00 2024-09-21T00:48:13.240 INFO:teuthology.orchestra.run.smithi150.stdout:(8/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 3.8 MB/s | 5.0 MB 00:01 2024-09-21T00:48:13.298 INFO:teuthology.orchestra.run.smithi150.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 433 kB/s | 25 kB 00:00 2024-09-21T00:48:13.341 INFO:teuthology.orchestra.run.smithi160.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 204 kB/s | 80 kB 00:00 2024-09-21T00:48:13.400 INFO:teuthology.orchestra.run.smithi160.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 740 kB/s | 191 kB 00:00 2024-09-21T00:48:13.433 INFO:teuthology.orchestra.run.smithi160.stdout:(42/157): libgfortran-11.5.0-2.el9.x86_64.rpm 1.7 MB/s | 801 kB 00:00 2024-09-21T00:48:13.459 INFO:teuthology.orchestra.run.smithi160.stdout:(43/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-21T00:48:13.484 INFO:teuthology.orchestra.run.smithi160.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T00:48:13.517 INFO:teuthology.orchestra.run.smithi160.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-21T00:48:13.543 INFO:teuthology.orchestra.run.smithi160.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-21T00:48:13.576 INFO:teuthology.orchestra.run.smithi160.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-09-21T00:48:13.619 INFO:teuthology.orchestra.run.smithi160.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-09-21T00:48:13.719 INFO:teuthology.orchestra.run.smithi160.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 107 kB 00:00 2024-09-21T00:48:13.752 INFO:teuthology.orchestra.run.smithi160.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-09-21T00:48:13.769 INFO:teuthology.orchestra.run.smithi160.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 432 kB/s | 22 kB 00:00 2024-09-21T00:48:13.803 INFO:teuthology.orchestra.run.smithi160.stdout:(52/157): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-09-21T00:48:13.828 INFO:teuthology.orchestra.run.smithi160.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-21T00:48:13.843 INFO:teuthology.orchestra.run.smithi039.stdout:(8/157): ceph-radosgw-19.3.0-5120.g18432218.el9 16 MB/s | 10 MB 00:00 2024-09-21T00:48:13.862 INFO:teuthology.orchestra.run.smithi160.stdout:(54/157): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-09-21T00:48:13.887 INFO:teuthology.orchestra.run.smithi160.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-09-21T00:48:13.893 INFO:teuthology.orchestra.run.smithi039.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 506 kB/s | 25 kB 00:00 2024-09-21T00:48:13.912 INFO:teuthology.orchestra.run.smithi160.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-09-21T00:48:13.924 INFO:teuthology.orchestra.run.smithi150.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 12 MB/s | 10 MB 00:00 2024-09-21T00:48:13.954 INFO:teuthology.orchestra.run.smithi160.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.5 MB/s | 171 kB 00:00 2024-09-21T00:48:13.990 INFO:teuthology.orchestra.run.smithi150.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.3 MB/s | 226 kB 00:00 2024-09-21T00:48:13.996 INFO:teuthology.orchestra.run.smithi160.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-09-21T00:48:14.021 INFO:teuthology.orchestra.run.smithi160.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-09-21T00:48:14.049 INFO:teuthology.orchestra.run.smithi150.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 464 kB/s | 27 kB 00:00 2024-09-21T00:48:14.080 INFO:teuthology.orchestra.run.smithi160.stdout:(60/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-09-21T00:48:14.105 INFO:teuthology.orchestra.run.smithi160.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-09-21T00:48:14.130 INFO:teuthology.orchestra.run.smithi160.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 488 kB/s | 24 kB 00:00 2024-09-21T00:48:14.133 INFO:teuthology.orchestra.run.smithi150.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 381 kB/s | 32 kB 00:00 2024-09-21T00:48:14.156 INFO:teuthology.orchestra.run.smithi160.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-09-21T00:48:14.181 INFO:teuthology.orchestra.run.smithi160.stdout:(64/157): libtool-2.4.6-46.el9.x86_64.rpm 3.6 MB/s | 581 kB 00:00 2024-09-21T00:48:14.206 INFO:teuthology.orchestra.run.smithi160.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-21T00:48:14.231 INFO:teuthology.orchestra.run.smithi160.stdout:(66/157): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-09-21T00:48:14.257 INFO:teuthology.orchestra.run.smithi160.stdout:(67/157): openblas-0.3.26-2.el9.x86_64.rpm 510 kB/s | 38 kB 00:00 2024-09-21T00:48:14.258 INFO:teuthology.orchestra.run.smithi150.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.0 MB/s | 766 kB 00:00 2024-09-21T00:48:14.333 INFO:teuthology.orchestra.run.smithi150.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 2.1 MB/s | 158 kB 00:00 2024-09-21T00:48:14.382 INFO:teuthology.orchestra.run.smithi160.stdout:(68/157): openblas-openmp-0.3.26-2.el9.x86_64.r 33 MB/s | 4.9 MB 00:00 2024-09-21T00:48:14.416 INFO:teuthology.orchestra.run.smithi160.stdout:(69/157): postgresql-13.16-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-21T00:48:14.417 INFO:teuthology.orchestra.run.smithi150.stdout:(16/157): librados-devel-19.3.0-5120.g18432218. 1.5 MB/s | 127 kB 00:00 2024-09-21T00:48:14.449 INFO:teuthology.orchestra.run.smithi160.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 13 MB/s | 910 kB 00:00 2024-09-21T00:48:14.474 INFO:teuthology.orchestra.run.smithi160.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-09-21T00:48:14.541 INFO:teuthology.orchestra.run.smithi160.stdout:(72/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-21T00:48:14.559 INFO:teuthology.orchestra.run.smithi150.stdout:(17/157): libradosstriper1-19.3.0-5120.g1843221 3.5 MB/s | 507 kB 00:00 2024-09-21T00:48:14.575 INFO:teuthology.orchestra.run.smithi160.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-09-21T00:48:14.600 INFO:teuthology.orchestra.run.smithi160.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-09-21T00:48:14.684 INFO:teuthology.orchestra.run.smithi160.stdout:(75/157): postgresql-server-13.16-1.el9.x86_64. 25 MB/s | 5.8 MB 00:00 2024-09-21T00:48:14.709 INFO:teuthology.orchestra.run.smithi160.stdout:(76/157): python3-colorama-0.4.6-3.el9.noarch.r 564 kB/s | 61 kB 00:00 2024-09-21T00:48:14.734 INFO:teuthology.orchestra.run.smithi160.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-21T00:48:14.759 INFO:teuthology.orchestra.run.smithi160.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-09-21T00:48:14.784 INFO:teuthology.orchestra.run.smithi160.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-09-21T00:48:14.810 INFO:teuthology.orchestra.run.smithi160.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-09-21T00:48:14.835 INFO:teuthology.orchestra.run.smithi160.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-21T00:48:14.859 INFO:teuthology.orchestra.run.smithi150.stdout:(18/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 7.6 MB/s | 18 MB 00:02 2024-09-21T00:48:14.860 INFO:teuthology.orchestra.run.smithi160.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-09-21T00:48:14.944 INFO:teuthology.orchestra.run.smithi160.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-21T00:48:14.951 INFO:teuthology.orchestra.run.smithi150.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 493 kB/s | 45 kB 00:00 2024-09-21T00:48:14.969 INFO:teuthology.orchestra.run.smithi160.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.2 MB/s | 250 kB 00:00 2024-09-21T00:48:14.995 INFO:teuthology.orchestra.run.smithi160.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-21T00:48:15.020 INFO:teuthology.orchestra.run.smithi160.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-21T00:48:15.035 INFO:teuthology.orchestra.run.smithi150.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 1.7 MB/s | 148 kB 00:00 2024-09-21T00:48:15.037 INFO:teuthology.orchestra.run.smithi160.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-09-21T00:48:15.062 INFO:teuthology.orchestra.run.smithi160.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 6.3 MB/s | 269 kB 00:00 2024-09-21T00:48:15.085 INFO:teuthology.orchestra.run.smithi150.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 3.2 MB/s | 161 kB 00:00 2024-09-21T00:48:15.088 INFO:teuthology.orchestra.run.smithi160.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-21T00:48:15.113 INFO:teuthology.orchestra.run.smithi160.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-21T00:48:15.144 INFO:teuthology.orchestra.run.smithi150.stdout:(22/157): python3-rados-19.3.0-5120.g18432218.e 5.4 MB/s | 321 kB 00:00 2024-09-21T00:48:15.146 INFO:teuthology.orchestra.run.smithi160.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 874 kB/s | 51 kB 00:00 2024-09-21T00:48:15.172 INFO:teuthology.orchestra.run.smithi160.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 919 kB/s | 54 kB 00:00 2024-09-21T00:48:15.219 INFO:teuthology.orchestra.run.smithi150.stdout:(23/157): librgw2-19.3.0-5120.g18432218.el9.x86 8.0 MB/s | 5.3 MB 00:00 2024-09-21T00:48:15.239 INFO:teuthology.orchestra.run.smithi160.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-09-21T00:48:15.245 INFO:teuthology.orchestra.run.smithi150.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 3.0 MB/s | 307 kB 00:00 2024-09-21T00:48:15.264 INFO:teuthology.orchestra.run.smithi160.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-09-21T00:48:15.270 INFO:teuthology.orchestra.run.smithi150.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.9 MB/s | 99 kB 00:00 2024-09-21T00:48:15.289 INFO:teuthology.orchestra.run.smithi160.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-09-21T00:48:15.295 INFO:teuthology.orchestra.run.smithi150.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.7 MB/s | 86 kB 00:00 2024-09-21T00:48:15.323 INFO:teuthology.orchestra.run.smithi160.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-21T00:48:15.346 INFO:teuthology.orchestra.run.smithi150.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-21T00:48:15.348 INFO:teuthology.orchestra.run.smithi160.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 612 kB/s | 36 kB 00:00 2024-09-21T00:48:15.373 INFO:teuthology.orchestra.run.smithi160.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-09-21T00:48:15.396 INFO:teuthology.orchestra.run.smithi150.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 685 kB/s | 34 kB 00:00 2024-09-21T00:48:15.398 INFO:teuthology.orchestra.run.smithi160.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-21T00:48:15.410 INFO:teuthology.orchestra.run.smithi039.stdout:(10/157): ceph-test-19.3.0-5120.g18432218.el9.x 26 MB/s | 40 MB 00:01 2024-09-21T00:48:15.424 INFO:teuthology.orchestra.run.smithi160.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T00:48:15.446 INFO:teuthology.orchestra.run.smithi150.stdout:(29/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 3.0 MB/s | 154 kB 00:00 2024-09-21T00:48:15.449 INFO:teuthology.orchestra.run.smithi160.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-21T00:48:15.460 INFO:teuthology.orchestra.run.smithi039.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 4.5 MB/s | 226 kB 00:00 2024-09-21T00:48:15.474 INFO:teuthology.orchestra.run.smithi160.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T00:48:15.496 INFO:teuthology.orchestra.run.smithi150.stdout:(30/157): rbd-mirror-19.3.0-5120.g18432218.el9. 14 MB/s | 3.1 MB 00:00 2024-09-21T00:48:15.499 INFO:teuthology.orchestra.run.smithi160.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-09-21T00:48:15.502 INFO:teuthology.orchestra.run.smithi039.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 653 kB/s | 27 kB 00:00 2024-09-21T00:48:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 634 kB/s | 32 kB 00:00 2024-09-21T00:48:15.575 INFO:teuthology.orchestra.run.smithi160.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-09-21T00:48:15.625 INFO:teuthology.orchestra.run.smithi160.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-21T00:48:15.636 INFO:teuthology.orchestra.run.smithi039.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 9.0 MB/s | 766 kB 00:00 2024-09-21T00:48:15.651 INFO:teuthology.orchestra.run.smithi160.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 12 MB/s | 308 kB 00:00 2024-09-21T00:48:15.675 INFO:teuthology.orchestra.run.smithi160.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-21T00:48:15.695 INFO:teuthology.orchestra.run.smithi039.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 2.7 MB/s | 158 kB 00:00 2024-09-21T00:48:15.745 INFO:teuthology.orchestra.run.smithi039.stdout:(16/157): librados-devel-19.3.0-5120.g18432218. 2.5 MB/s | 127 kB 00:00 2024-09-21T00:48:15.834 INFO:teuthology.orchestra.run.smithi160.stdout:(108/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-09-21T00:48:15.868 INFO:teuthology.orchestra.run.smithi160.stdout:(109/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-09-21T00:48:15.893 INFO:teuthology.orchestra.run.smithi160.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-09-21T00:48:15.918 INFO:teuthology.orchestra.run.smithi160.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-09-21T00:48:15.937 INFO:teuthology.orchestra.run.smithi039.stdout:(17/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 5.9 MB/s | 18 MB 00:02 2024-09-21T00:48:15.943 INFO:teuthology.orchestra.run.smithi160.stdout:(112/157): liboath-2.6.7-2.el9.x86_64.rpm 660 kB/s | 49 kB 00:00 2024-09-21T00:48:15.963 INFO:teuthology.orchestra.run.smithi150.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 16 MB/s | 7.4 MB 00:00 2024-09-21T00:48:15.969 INFO:teuthology.orchestra.run.smithi160.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-09-21T00:48:15.971 INFO:teuthology.orchestra.run.smithi039.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 2.2 MB/s | 507 kB 00:00 2024-09-21T00:48:16.011 INFO:teuthology.orchestra.run.smithi160.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-21T00:48:16.014 INFO:teuthology.orchestra.run.smithi150.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 5.7 MB/s | 292 kB 00:00 2024-09-21T00:48:16.036 INFO:teuthology.orchestra.run.smithi160.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-21T00:48:16.061 INFO:teuthology.orchestra.run.smithi160.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-21T00:48:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:(33/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 991 kB/s | 49 kB 00:00 2024-09-21T00:48:16.087 INFO:teuthology.orchestra.run.smithi160.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 563 kB/s | 14 kB 00:00 2024-09-21T00:48:16.106 INFO:teuthology.orchestra.run.smithi150.stdout:(34/157): ceph-prometheus-alerts-19.3.0-5120.g1 395 kB/s | 16 kB 00:00 2024-09-21T00:48:16.120 INFO:teuthology.orchestra.run.smithi160.stdout:(118/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.6 MB/s | 839 kB 00:00 2024-09-21T00:48:16.154 INFO:teuthology.orchestra.run.smithi160.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-09-21T00:48:16.156 INFO:teuthology.orchestra.run.smithi150.stdout:(35/157): ceph-volume-19.3.0-5120.g18432218.el9 5.6 MB/s | 285 kB 00:00 2024-09-21T00:48:16.187 INFO:teuthology.orchestra.run.smithi160.stdout:(120/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 8.9 MB/s | 303 kB 00:00 2024-09-21T00:48:16.196 INFO:teuthology.orchestra.run.smithi039.stdout:(19/157): ceph-common-19.3.0-5120.g18432218.el9 3.7 MB/s | 20 MB 00:05 2024-09-21T00:48:16.213 INFO:teuthology.orchestra.run.smithi039.stdout:(20/157): python3-ceph-argparse-19.3.0-5120.g18 187 kB/s | 45 kB 00:00 2024-09-21T00:48:16.213 INFO:teuthology.orchestra.run.smithi160.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-21T00:48:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:(36/157): cephadm-19.3.0-5120.g18432218.el9.noa 10 MB/s | 784 kB 00:00 2024-09-21T00:48:16.238 INFO:teuthology.orchestra.run.smithi160.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-21T00:48:16.263 INFO:teuthology.orchestra.run.smithi039.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 3.1 MB/s | 161 kB 00:00 2024-09-21T00:48:16.282 INFO:teuthology.orchestra.run.smithi150.stdout:(37/157): cephfs-top-19.3.0-5120.g18432218.el9. 369 kB/s | 18 kB 00:00 2024-09-21T00:48:16.289 INFO:teuthology.orchestra.run.smithi039.stdout:(22/157): python3-ceph-common-19.3.0-5120.g1843 1.6 MB/s | 148 kB 00:00 2024-09-21T00:48:16.297 INFO:teuthology.orchestra.run.smithi160.stdout:(123/157): libarrow-9.0.0-11.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-09-21T00:48:16.322 INFO:teuthology.orchestra.run.smithi160.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-09-21T00:48:16.347 INFO:teuthology.orchestra.run.smithi160.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-21T00:48:16.356 INFO:teuthology.orchestra.run.smithi039.stdout:(23/157): python3-rados-19.3.0-5120.g18432218.e 3.4 MB/s | 321 kB 00:00 2024-09-21T00:48:16.372 INFO:teuthology.orchestra.run.smithi160.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-09-21T00:48:16.398 INFO:teuthology.orchestra.run.smithi039.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 2.8 MB/s | 307 kB 00:00 2024-09-21T00:48:16.398 INFO:teuthology.orchestra.run.smithi160.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-09-21T00:48:16.423 INFO:teuthology.orchestra.run.smithi039.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.5 MB/s | 99 kB 00:00 2024-09-21T00:48:16.423 INFO:teuthology.orchestra.run.smithi160.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-21T00:48:16.424 INFO:teuthology.orchestra.run.smithi150.stdout:(38/157): fuse-2.9.9-16.el9.x86_64.rpm 563 kB/s | 80 kB 00:00 2024-09-21T00:48:16.448 INFO:teuthology.orchestra.run.smithi160.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-09-21T00:48:16.465 INFO:teuthology.orchestra.run.smithi039.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.3 MB/s | 86 kB 00:00 2024-09-21T00:48:16.473 INFO:teuthology.orchestra.run.smithi160.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T00:48:16.507 INFO:teuthology.orchestra.run.smithi160.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-21T00:48:16.540 INFO:teuthology.orchestra.run.smithi039.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 2.2 MB/s | 170 kB 00:00 2024-09-21T00:48:16.541 INFO:teuthology.orchestra.run.smithi160.stdout:(132/157): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-09-21T00:48:16.549 INFO:teuthology.orchestra.run.smithi150.stdout:(39/157): libgfortran-11.5.0-2.el9.x86_64.rpm 6.3 MB/s | 801 kB 00:00 2024-09-21T00:48:16.566 INFO:teuthology.orchestra.run.smithi160.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-09-21T00:48:16.591 INFO:teuthology.orchestra.run.smithi160.stdout:(134/157): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-21T00:48:16.609 INFO:teuthology.orchestra.run.smithi039.stdout:(28/157): rbd-mirror-19.3.0-5120.g18432218.el9. 17 MB/s | 3.1 MB 00:00 2024-09-21T00:48:16.617 INFO:teuthology.orchestra.run.smithi160.stdout:(135/157): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-09-21T00:48:16.632 INFO:teuthology.orchestra.run.smithi039.stdout:(29/157): ceph-grafana-dashboards-19.3.0-5120.g 373 kB/s | 34 kB 00:00 2024-09-21T00:48:16.642 INFO:teuthology.orchestra.run.smithi160.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-21T00:48:16.657 INFO:teuthology.orchestra.run.smithi039.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 3.0 MB/s | 154 kB 00:00 2024-09-21T00:48:16.667 INFO:teuthology.orchestra.run.smithi160.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-09-21T00:48:16.691 INFO:teuthology.orchestra.run.smithi150.stdout:(40/157): ceph-mgr-dashboard-19.3.0-5120.g18432 11 MB/s | 14 MB 00:01 2024-09-21T00:48:16.692 INFO:teuthology.orchestra.run.smithi160.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-09-21T00:48:16.716 INFO:teuthology.orchestra.run.smithi150.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T00:48:16.734 INFO:teuthology.orchestra.run.smithi160.stdout:(139/157): python3-grpcio-1.46.7-10.el9.x86_64. 3.9 MB/s | 2.0 MB 00:00 2024-09-21T00:48:16.758 INFO:teuthology.orchestra.run.smithi039.stdout:(31/157): librgw2-19.3.0-5120.g18432218.el9.x86 6.4 MB/s | 5.3 MB 00:00 2024-09-21T00:48:16.760 INFO:teuthology.orchestra.run.smithi160.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-09-21T00:48:16.785 INFO:teuthology.orchestra.run.smithi160.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-09-21T00:48:16.810 INFO:teuthology.orchestra.run.smithi160.stdout:(142/157): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-21T00:48:16.816 INFO:teuthology.orchestra.run.smithi150.stdout:(42/157): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-09-21T00:48:16.835 INFO:teuthology.orchestra.run.smithi160.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 990 kB/s | 74 kB 00:00 2024-09-21T00:48:16.850 INFO:teuthology.orchestra.run.smithi039.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 3.1 MB/s | 292 kB 00:00 2024-09-21T00:48:16.861 INFO:teuthology.orchestra.run.smithi160.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-21T00:48:16.867 INFO:teuthology.orchestra.run.smithi150.stdout:(43/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-21T00:48:16.886 INFO:teuthology.orchestra.run.smithi160.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-21T00:48:16.892 INFO:teuthology.orchestra.run.smithi150.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-21T00:48:16.903 INFO:teuthology.orchestra.run.smithi160.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 479 kB/s | 20 kB 00:00 2024-09-21T00:48:16.928 INFO:teuthology.orchestra.run.smithi160.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-21T00:48:16.951 INFO:teuthology.orchestra.run.smithi150.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-09-21T00:48:16.953 INFO:teuthology.orchestra.run.smithi160.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-09-21T00:48:16.958 INFO:teuthology.orchestra.run.smithi039.stdout:(33/157): ceph-mgr-diskprediction-local-19.3.0- 25 MB/s | 7.4 MB 00:00 2024-09-21T00:48:16.976 INFO:teuthology.orchestra.run.smithi150.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-21T00:48:16.984 INFO:teuthology.orchestra.run.smithi039.stdout:(34/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 369 kB/s | 49 kB 00:00 2024-09-21T00:48:17.021 INFO:teuthology.orchestra.run.smithi160.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-21T00:48:17.030 INFO:teuthology.orchestra.run.smithi039.stdout:(35/157): ceph-prometheus-alerts-19.3.0-5120.g1 232 kB/s | 16 kB 00:00 2024-09-21T00:48:17.035 INFO:teuthology.orchestra.run.smithi150.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-21T00:48:17.068 INFO:teuthology.orchestra.run.smithi039.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 3.3 MB/s | 285 kB 00:00 2024-09-21T00:48:17.101 INFO:teuthology.orchestra.run.smithi039.stdout:(37/157): cephadm-19.3.0-5120.g18432218.el9.noa 11 MB/s | 784 kB 00:00 2024-09-21T00:48:17.127 INFO:teuthology.orchestra.run.smithi039.stdout:(38/157): cephfs-top-19.3.0-5120.g18432218.el9. 315 kB/s | 18 kB 00:00 2024-09-21T00:48:17.187 INFO:teuthology.orchestra.run.smithi160.stdout:(150/157): librados2-19.3.0-5120.g18432218.el9. 13 MB/s | 3.4 MB 00:00 2024-09-21T00:48:17.302 INFO:teuthology.orchestra.run.smithi039.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 398 kB/s | 80 kB 00:00 2024-09-21T00:48:17.302 INFO:teuthology.orchestra.run.smithi150.stdout:(48/157): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-21T00:48:17.329 INFO:teuthology.orchestra.run.smithi160.stdout:(151/157): librbd1-19.3.0-5120.g18432218.el9.x8 8.4 MB/s | 3.2 MB 00:00 2024-09-21T00:48:17.361 INFO:teuthology.orchestra.run.smithi150.stdout:(49/157): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-09-21T00:48:17.380 INFO:teuthology.orchestra.run.smithi160.stdout:(152/157): NetworkManager-1.51.0-1.el9.x86_64.r 6.5 MB/s | 2.3 MB 00:00 2024-09-21T00:48:17.386 INFO:teuthology.orchestra.run.smithi150.stdout:(50/157): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 107 kB 00:00 2024-09-21T00:48:17.411 INFO:teuthology.orchestra.run.smithi150.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 432 kB/s | 22 kB 00:00 2024-09-21T00:48:17.427 INFO:teuthology.orchestra.run.smithi039.stdout:(40/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-21T00:48:17.438 INFO:teuthology.orchestra.run.smithi160.stdout:(153/157): NetworkManager-tui-1.51.0-1.el9.x86_ 4.2 MB/s | 249 kB 00:00 2024-09-21T00:48:17.452 INFO:teuthology.orchestra.run.smithi039.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-21T00:48:17.453 INFO:teuthology.orchestra.run.smithi150.stdout:(52/157): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-09-21T00:48:17.478 INFO:teuthology.orchestra.run.smithi150.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-09-21T00:48:17.486 INFO:teuthology.orchestra.run.smithi039.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-09-21T00:48:17.488 INFO:teuthology.orchestra.run.smithi160.stdout:(154/157): NetworkManager-team-1.51.0-1.el9.x86 258 kB/s | 41 kB 00:00 2024-09-21T00:48:17.537 INFO:teuthology.orchestra.run.smithi150.stdout:(54/157): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-09-21T00:48:17.539 INFO:teuthology.orchestra.run.smithi160.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-09-21T00:48:17.544 INFO:teuthology.orchestra.run.smithi039.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-21T00:48:17.596 INFO:teuthology.orchestra.run.smithi150.stdout:(55/157): libaio-devel-0.3.111-13.el9.x86_64.rp 190 kB/s | 11 kB 00:00 2024-09-21T00:48:17.611 INFO:teuthology.orchestra.run.smithi039.stdout:(44/157): python3-cryptography-36.0.1-4.el9.x86 7.7 MB/s | 1.2 MB 00:00 2024-09-21T00:48:17.645 INFO:teuthology.orchestra.run.smithi039.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-09-21T00:48:17.663 INFO:teuthology.orchestra.run.smithi150.stdout:(56/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.5 MB/s | 171 kB 00:00 2024-09-21T00:48:17.689 INFO:teuthology.orchestra.run.smithi160.stdout:(156/157): NetworkManager-libnm-1.51.0-1.el9.x8 3.6 MB/s | 1.8 MB 00:00 2024-09-21T00:48:17.721 INFO:teuthology.orchestra.run.smithi039.stdout:(46/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-09-21T00:48:17.721 INFO:teuthology.orchestra.run.smithi150.stdout:(57/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-09-21T00:48:17.746 INFO:teuthology.orchestra.run.smithi039.stdout:(47/157): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-09-21T00:48:17.771 INFO:teuthology.orchestra.run.smithi150.stdout:(58/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-09-21T00:48:17.830 INFO:teuthology.orchestra.run.smithi150.stdout:(59/157): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-09-21T00:48:17.888 INFO:teuthology.orchestra.run.smithi039.stdout:(48/157): automake-1.16.2-8.el9.noarch.rpm 4.0 MB/s | 689 kB 00:00 2024-09-21T00:48:17.930 INFO:teuthology.orchestra.run.smithi039.stdout:(49/157): bison-3.7.4-5.el9.x86_64.rpm 5.0 MB/s | 943 kB 00:00 2024-09-21T00:48:17.955 INFO:teuthology.orchestra.run.smithi039.stdout:(50/157): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-09-21T00:48:17.972 INFO:teuthology.orchestra.run.smithi039.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 519 kB/s | 22 kB 00:00 2024-09-21T00:48:18.014 INFO:teuthology.orchestra.run.smithi039.stdout:(52/157): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-09-21T00:48:18.056 INFO:teuthology.orchestra.run.smithi039.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 393 kB/s | 33 kB 00:00 2024-09-21T00:48:18.089 INFO:teuthology.orchestra.run.smithi039.stdout:(54/157): flexiblas-openblas-openmp-3.0.4-8.el9 549 kB/s | 18 kB 00:00 2024-09-21T00:48:18.155 INFO:teuthology.orchestra.run.smithi150.stdout:(60/157): libtool-2.4.6-46.el9.x86_64.rpm 1.7 MB/s | 581 kB 00:00 2024-09-21T00:48:18.157 INFO:teuthology.orchestra.run.smithi039.stdout:(55/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-09-21T00:48:18.173 INFO:teuthology.orchestra.run.smithi039.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 134 kB/s | 11 kB 00:00 2024-09-21T00:48:18.199 INFO:teuthology.orchestra.run.smithi039.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 4.0 MB/s | 171 kB 00:00 2024-09-21T00:48:18.214 INFO:teuthology.orchestra.run.smithi150.stdout:(61/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 420 kB/s | 24 kB 00:00 2024-09-21T00:48:18.224 INFO:teuthology.orchestra.run.smithi039.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-21T00:48:18.231 INFO:teuthology.orchestra.run.smithi160.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.2 MB/s | 2.4 MB 00:00 2024-09-21T00:48:18.238 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T00:48:18.238 INFO:teuthology.orchestra.run.smithi160.stdout:Total 24 MB/s | 228 MB 00:09 2024-09-21T00:48:18.241 INFO:teuthology.orchestra.run.smithi039.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-21T00:48:18.272 INFO:teuthology.orchestra.run.smithi150.stdout:(62/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.6 MB/s | 3.0 MB 00:00 2024-09-21T00:48:18.275 INFO:teuthology.orchestra.run.smithi039.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-09-21T00:48:18.298 INFO:teuthology.orchestra.run.smithi150.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-09-21T00:48:18.309 INFO:teuthology.orchestra.run.smithi039.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-09-21T00:48:18.325 INFO:teuthology.orchestra.run.smithi039.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 490 kB/s | 24 kB 00:00 2024-09-21T00:48:18.331 INFO:teuthology.orchestra.run.smithi150.stdout:(64/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-21T00:48:18.342 INFO:teuthology.orchestra.run.smithi039.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-21T00:48:18.357 INFO:teuthology.orchestra.run.smithi150.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-21T00:48:18.385 INFO:teuthology.orchestra.run.smithi039.stdout:(64/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 292 kB 00:00 2024-09-21T00:48:18.407 INFO:teuthology.orchestra.run.smithi150.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 512 kB/s | 38 kB 00:00 2024-09-21T00:48:18.418 INFO:teuthology.orchestra.run.smithi039.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-09-21T00:48:18.443 INFO:teuthology.orchestra.run.smithi039.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 669 kB/s | 38 kB 00:00 2024-09-21T00:48:18.624 INFO:teuthology.orchestra.run.smithi150.stdout:(67/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-09-21T00:48:18.660 INFO:teuthology.orchestra.run.smithi039.stdout:(67/157): ceph-mgr-dashboard-19.3.0-5120.g18432 7.0 MB/s | 14 MB 00:02 2024-09-21T00:48:18.778 INFO:teuthology.orchestra.run.smithi039.stdout:(68/157): openblas-openmp-0.3.26-2.el9.x86_64.r 14 MB/s | 4.9 MB 00:00 2024-09-21T00:48:18.947 INFO:teuthology.orchestra.run.smithi039.stdout:(69/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T00:48:18.947 INFO:teuthology.orchestra.run.smithi039.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 8.9 MB/s | 910 kB 00:00 2024-09-21T00:48:18.947 INFO:teuthology.orchestra.run.smithi039.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-21T00:48:18.974 INFO:teuthology.orchestra.run.smithi150.stdout:(68/157): ceph-test-19.3.0-5120.g18432218.el9.x 7.0 MB/s | 40 MB 00:05 2024-09-21T00:48:18.986 INFO:teuthology.orchestra.run.smithi039.stdout:(72/157): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-21T00:48:19.037 INFO:teuthology.orchestra.run.smithi039.stdout:(73/157): postgresql-13.16-1.el9.x86_64.rpm 4.2 MB/s | 1.6 MB 00:00 2024-09-21T00:48:19.050 INFO:teuthology.orchestra.run.smithi150.stdout:(69/157): openblas-openmp-0.3.26-2.el9.x86_64.r 7.1 MB/s | 4.9 MB 00:00 2024-09-21T00:48:19.091 INFO:teuthology.orchestra.run.smithi150.stdout:(70/157): postgresql-13.16-1.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-09-21T00:48:19.100 INFO:teuthology.orchestra.run.smithi039.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 774 kB/s | 87 kB 00:00 2024-09-21T00:48:19.117 INFO:teuthology.orchestra.run.smithi150.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-21T00:48:19.196 INFO:teuthology.orchestra.run.smithi039.stdout:(75/157): postgresql-server-13.16-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-09-21T00:48:19.200 INFO:teuthology.orchestra.run.smithi150.stdout:(72/157): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-09-21T00:48:19.221 INFO:teuthology.orchestra.run.smithi039.stdout:(76/157): python3-colorama-0.4.6-3.el9.noarch.r 507 kB/s | 61 kB 00:00 2024-09-21T00:48:19.255 INFO:teuthology.orchestra.run.smithi039.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-21T00:48:19.276 INFO:teuthology.orchestra.run.smithi150.stdout:(73/157): postgresql-server-13.16-1.el9.x86_64. 31 MB/s | 5.8 MB 00:00 2024-09-21T00:48:19.280 INFO:teuthology.orchestra.run.smithi039.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-09-21T00:48:19.301 INFO:teuthology.orchestra.run.smithi150.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 872 kB/s | 87 kB 00:00 2024-09-21T00:48:19.305 INFO:teuthology.orchestra.run.smithi039.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-09-21T00:48:19.331 INFO:teuthology.orchestra.run.smithi039.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-21T00:48:19.364 INFO:teuthology.orchestra.run.smithi039.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-21T00:48:19.381 INFO:teuthology.orchestra.run.smithi039.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-09-21T00:48:19.385 INFO:teuthology.orchestra.run.smithi150.stdout:(75/157): postgresql-contrib-13.16-1.el9.x86_64 2.2 MB/s | 910 kB 00:00 2024-09-21T00:48:19.410 INFO:teuthology.orchestra.run.smithi150.stdout:(76/157): python3-colorama-0.4.6-3.el9.noarch.r 562 kB/s | 61 kB 00:00 2024-09-21T00:48:19.444 INFO:teuthology.orchestra.run.smithi150.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-21T00:48:19.489 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T00:48:19.498 INFO:teuthology.orchestra.run.smithi039.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-21T00:48:19.523 INFO:teuthology.orchestra.run.smithi039.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-09-21T00:48:19.527 INFO:teuthology.orchestra.run.smithi150.stdout:(78/157): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 6.0 MB 00:00 2024-09-21T00:48:19.545 INFO:teuthology.orchestra.run.smithi150.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-09-21T00:48:19.549 INFO:teuthology.orchestra.run.smithi039.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-21T00:48:19.570 INFO:teuthology.orchestra.run.smithi150.stdout:(80/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-09-21T00:48:19.574 INFO:teuthology.orchestra.run.smithi039.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-21T00:48:19.586 INFO:teuthology.orchestra.run.smithi150.stdout:(81/157): python3-jsonpointer-2.0-4.el9.noarch. 326 kB/s | 19 kB 00:00 2024-09-21T00:48:19.591 INFO:teuthology.orchestra.run.smithi039.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-09-21T00:48:19.612 INFO:teuthology.orchestra.run.smithi150.stdout:(82/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-21T00:48:19.624 INFO:teuthology.orchestra.run.smithi039.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-09-21T00:48:19.637 INFO:teuthology.orchestra.run.smithi150.stdout:(83/157): python3-markupsafe-1.1.1-12.el9.x86_6 517 kB/s | 35 kB 00:00 2024-09-21T00:48:19.643 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T00:48:19.644 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T00:48:19.650 INFO:teuthology.orchestra.run.smithi039.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-09-21T00:48:19.675 INFO:teuthology.orchestra.run.smithi039.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-21T00:48:19.679 INFO:teuthology.orchestra.run.smithi150.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-09-21T00:48:19.692 INFO:teuthology.orchestra.run.smithi039.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-21T00:48:19.704 INFO:teuthology.orchestra.run.smithi150.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-09-21T00:48:19.709 INFO:teuthology.orchestra.run.smithi039.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-09-21T00:48:19.788 INFO:teuthology.orchestra.run.smithi150.stdout:(86/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-09-21T00:48:19.809 INFO:teuthology.orchestra.run.smithi039.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-21T00:48:19.813 INFO:teuthology.orchestra.run.smithi150.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 389 kB/s | 42 kB 00:00 2024-09-21T00:48:19.826 INFO:teuthology.orchestra.run.smithi039.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-09-21T00:48:19.838 INFO:teuthology.orchestra.run.smithi150.stdout:(88/157): python3-packaging-20.9-5.el9.noarch.r 484 kB/s | 77 kB 00:00 2024-09-21T00:48:19.843 INFO:teuthology.orchestra.run.smithi039.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-09-21T00:48:19.877 INFO:teuthology.orchestra.run.smithi039.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-21T00:48:19.894 INFO:teuthology.orchestra.run.smithi039.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 713 kB/s | 36 kB 00:00 2024-09-21T00:48:19.905 INFO:teuthology.orchestra.run.smithi150.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-09-21T00:48:19.919 INFO:teuthology.orchestra.run.smithi039.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-21T00:48:19.931 INFO:teuthology.orchestra.run.smithi150.stdout:(90/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-21T00:48:19.944 INFO:teuthology.orchestra.run.smithi039.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-21T00:48:19.956 INFO:teuthology.orchestra.run.smithi150.stdout:(91/157): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-21T00:48:19.970 INFO:teuthology.orchestra.run.smithi039.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T00:48:19.981 INFO:teuthology.orchestra.run.smithi150.stdout:(92/157): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-21T00:48:20.003 INFO:teuthology.orchestra.run.smithi039.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-09-21T00:48:20.006 INFO:teuthology.orchestra.run.smithi150.stdout:(93/157): python3-requests-oauthlib-1.3.0-12.el 711 kB/s | 54 kB 00:00 2024-09-21T00:48:20.020 INFO:teuthology.orchestra.run.smithi039.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T00:48:20.037 INFO:teuthology.orchestra.run.smithi039.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-09-21T00:48:20.048 INFO:teuthology.orchestra.run.smithi150.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-09-21T00:48:20.074 INFO:teuthology.orchestra.run.smithi150.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 660 kB/s | 44 kB 00:00 2024-09-21T00:48:20.107 INFO:teuthology.orchestra.run.smithi150.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-21T00:48:20.112 INFO:teuthology.orchestra.run.smithi039.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-09-21T00:48:20.132 INFO:teuthology.orchestra.run.smithi150.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 611 kB/s | 36 kB 00:00 2024-09-21T00:48:20.166 INFO:teuthology.orchestra.run.smithi150.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-09-21T00:48:20.171 INFO:teuthology.orchestra.run.smithi039.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-21T00:48:20.191 INFO:teuthology.orchestra.run.smithi150.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-09-21T00:48:20.205 INFO:teuthology.orchestra.run.smithi039.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-21T00:48:20.230 INFO:teuthology.orchestra.run.smithi039.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 786 kB/s | 19 kB 00:00 2024-09-21T00:48:20.233 INFO:teuthology.orchestra.run.smithi150.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 864 kB/s | 58 kB 00:00 2024-09-21T00:48:20.263 INFO:teuthology.orchestra.run.smithi039.stdout:(108/157): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-09-21T00:48:20.267 INFO:teuthology.orchestra.run.smithi150.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.2 MB/s | 96 kB 00:00 2024-09-21T00:48:20.292 INFO:teuthology.orchestra.run.smithi150.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-21T00:48:20.314 INFO:teuthology.orchestra.run.smithi039.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-09-21T00:48:20.326 INFO:teuthology.orchestra.run.smithi150.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-09-21T00:48:20.339 INFO:teuthology.orchestra.run.smithi039.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T00:48:20.351 INFO:teuthology.orchestra.run.smithi150.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 379 kB/s | 22 kB 00:00 2024-09-21T00:48:20.365 INFO:teuthology.orchestra.run.smithi039.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T00:48:20.390 INFO:teuthology.orchestra.run.smithi039.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-09-21T00:48:20.457 INFO:teuthology.orchestra.run.smithi039.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-21T00:48:20.499 INFO:teuthology.orchestra.run.smithi039.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-21T00:48:20.524 INFO:teuthology.orchestra.run.smithi039.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-21T00:48:20.535 INFO:teuthology.orchestra.run.smithi150.stdout:(105/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-09-21T00:48:20.549 INFO:teuthology.orchestra.run.smithi039.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-21T00:48:20.575 INFO:teuthology.orchestra.run.smithi039.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 562 kB/s | 14 kB 00:00 2024-09-21T00:48:20.594 INFO:teuthology.orchestra.run.smithi150.stdout:(106/157): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-09-21T00:48:20.636 INFO:teuthology.orchestra.run.smithi150.stdout:(107/157): gperftools-libs-2.9.1-3.el9.x86_64.r 3.2 MB/s | 308 kB 00:00 2024-09-21T00:48:20.669 INFO:teuthology.orchestra.run.smithi150.stdout:(108/157): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-09-21T00:48:20.694 INFO:teuthology.orchestra.run.smithi150.stdout:(109/157): grpc-data-1.46.7-10.el9.noarch.rpm 197 kB/s | 19 kB 00:00 2024-09-21T00:48:20.720 INFO:teuthology.orchestra.run.smithi150.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-09-21T00:48:20.745 INFO:teuthology.orchestra.run.smithi150.stdout:(111/157): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-09-21T00:48:20.770 INFO:teuthology.orchestra.run.smithi150.stdout:(112/157): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T00:48:20.775 INFO:teuthology.orchestra.run.smithi039.stdout:(118/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-09-21T00:48:20.796 INFO:teuthology.orchestra.run.smithi150.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-09-21T00:48:20.817 INFO:teuthology.orchestra.run.smithi039.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 710 kB/s | 172 kB 00:00 2024-09-21T00:48:20.837 INFO:teuthology.orchestra.run.smithi150.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-21T00:48:20.851 INFO:teuthology.orchestra.run.smithi039.stdout:(120/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 9.5 MB/s | 303 kB 00:00 2024-09-21T00:48:20.863 INFO:teuthology.orchestra.run.smithi150.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-21T00:48:20.888 INFO:teuthology.orchestra.run.smithi150.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-21T00:48:20.892 INFO:teuthology.orchestra.run.smithi039.stdout:(121/157): python3-google-auth-2.34.0-1.el9.noa 5.2 MB/s | 220 kB 00:00 2024-09-21T00:48:20.913 INFO:teuthology.orchestra.run.smithi150.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 569 kB/s | 14 kB 00:00 2024-09-21T00:48:20.918 INFO:teuthology.orchestra.run.smithi039.stdout:(122/157): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-09-21T00:48:20.939 INFO:teuthology.orchestra.run.smithi150.stdout:(118/157): python3-asyncssh-2.13.2-5.el9.noarch 3.7 MB/s | 548 kB 00:00 2024-09-21T00:48:20.951 INFO:teuthology.orchestra.run.smithi039.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-21T00:48:20.964 INFO:teuthology.orchestra.run.smithi150.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-21T00:48:20.968 INFO:teuthology.orchestra.run.smithi039.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 649 kB/s | 11 kB 00:00 2024-09-21T00:48:20.985 INFO:teuthology.orchestra.run.smithi039.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 1.0 MB/s | 18 kB 00:00 2024-09-21T00:48:21.002 INFO:teuthology.orchestra.run.smithi039.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 1.4 MB/s | 23 kB 00:00 2024-09-21T00:48:21.006 INFO:teuthology.orchestra.run.smithi150.stdout:(120/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-09-21T00:48:21.019 INFO:teuthology.orchestra.run.smithi039.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 1.1 MB/s | 19 kB 00:00 2024-09-21T00:48:21.031 INFO:teuthology.orchestra.run.smithi150.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-21T00:48:21.036 INFO:teuthology.orchestra.run.smithi039.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 1.2 MB/s | 20 kB 00:00 2024-09-21T00:48:21.107 INFO:teuthology.orchestra.run.smithi150.stdout:(122/157): libarrow-9.0.0-11.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-09-21T00:48:21.129 INFO:teuthology.orchestra.run.smithi039.stdout:(129/157): libarrow-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-09-21T00:48:21.132 INFO:teuthology.orchestra.run.smithi150.stdout:(123/157): python3-google-auth-2.34.0-1.el9.noa 1.7 MB/s | 220 kB 00:00 2024-09-21T00:48:21.148 INFO:teuthology.orchestra.run.smithi150.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 652 kB/s | 11 kB 00:00 2024-09-21T00:48:21.153 INFO:teuthology.orchestra.run.smithi039.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-09-21T00:48:21.174 INFO:teuthology.orchestra.run.smithi150.stdout:(125/157): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-09-21T00:48:21.188 INFO:teuthology.orchestra.run.smithi039.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-21T00:48:21.191 INFO:teuthology.orchestra.run.smithi150.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 420 kB/s | 18 kB 00:00 2024-09-21T00:48:21.205 INFO:teuthology.orchestra.run.smithi039.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 3.4 MB/s | 58 kB 00:00 2024-09-21T00:48:21.207 INFO:teuthology.orchestra.run.smithi150.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 690 kB/s | 23 kB 00:00 2024-09-21T00:48:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 585 kB/s | 19 kB 00:00 2024-09-21T00:48:21.238 INFO:teuthology.orchestra.run.smithi039.stdout:(133/157): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-21T00:48:21.241 INFO:teuthology.orchestra.run.smithi150.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 584 kB/s | 20 kB 00:00 2024-09-21T00:48:21.255 INFO:teuthology.orchestra.run.smithi039.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 979 kB/s | 16 kB 00:00 2024-09-21T00:48:21.258 INFO:teuthology.orchestra.run.smithi150.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-09-21T00:48:21.281 INFO:teuthology.orchestra.run.smithi039.stdout:(135/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-21T00:48:21.284 INFO:teuthology.orchestra.run.smithi150.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T00:48:21.298 INFO:teuthology.orchestra.run.smithi039.stdout:(136/157): python3-pyperclip-1.8.2-2.el9.noarch 1.6 MB/s | 28 kB 00:00 2024-09-21T00:48:21.301 INFO:teuthology.orchestra.run.smithi150.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 3.4 MB/s | 58 kB 00:00 2024-09-21T00:48:21.331 INFO:teuthology.orchestra.run.smithi039.stdout:(137/157): python3-kubernetes-26.1.0-2.el9.noar 3.5 MB/s | 1.0 MB 00:00 2024-09-21T00:48:21.334 INFO:teuthology.orchestra.run.smithi150.stdout:(133/157): python3-kubernetes-26.1.0-2.el9.noar 9.3 MB/s | 1.0 MB 00:00 2024-09-21T00:48:21.348 INFO:teuthology.orchestra.run.smithi039.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-09-21T00:48:21.351 INFO:teuthology.orchestra.run.smithi150.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 986 kB/s | 16 kB 00:00 2024-09-21T00:48:21.390 INFO:teuthology.orchestra.run.smithi039.stdout:(139/157): python3-grpcio-1.46.7-10.el9.x86_64. 4.1 MB/s | 2.0 MB 00:00 2024-09-21T00:48:21.393 INFO:teuthology.orchestra.run.smithi150.stdout:(135/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-21T00:48:21.415 INFO:teuthology.orchestra.run.smithi039.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-09-21T00:48:21.419 INFO:teuthology.orchestra.run.smithi150.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-09-21T00:48:21.441 INFO:teuthology.orchestra.run.smithi039.stdout:(141/157): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-09-21T00:48:21.444 INFO:teuthology.orchestra.run.smithi150.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-09-21T00:48:21.463 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T00:48:21.465 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T00:48:21.469 INFO:teuthology.orchestra.run.smithi150.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-09-21T00:48:21.491 INFO:teuthology.orchestra.run.smithi039.stdout:(142/157): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-09-21T00:48:21.494 INFO:teuthology.orchestra.run.smithi150.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T00:48:21.516 INFO:teuthology.orchestra.run.smithi039.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 743 kB/s | 74 kB 00:00 2024-09-21T00:48:21.542 INFO:teuthology.orchestra.run.smithi039.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-21T00:48:21.545 INFO:teuthology.orchestra.run.smithi150.stdout:(140/157): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-09-21T00:48:21.570 INFO:teuthology.orchestra.run.smithi150.stdout:(141/157): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-09-21T00:48:21.575 INFO:teuthology.orchestra.run.smithi039.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-09-21T00:48:21.595 INFO:teuthology.orchestra.run.smithi150.stdout:(142/157): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-09-21T00:48:21.600 INFO:teuthology.orchestra.run.smithi039.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 343 kB/s | 20 kB 00:00 2024-09-21T00:48:21.621 INFO:teuthology.orchestra.run.smithi150.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 986 kB/s | 74 kB 00:00 2024-09-21T00:48:21.643 INFO:teuthology.orchestra.run.smithi039.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-21T00:48:21.646 INFO:teuthology.orchestra.run.smithi150.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-21T00:48:21.668 INFO:teuthology.orchestra.run.smithi039.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-09-21T00:48:21.672 INFO:teuthology.orchestra.run.smithi150.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T00:48:21.696 INFO:teuthology.orchestra.run.smithi150.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-09-21T00:48:21.722 INFO:teuthology.orchestra.run.smithi150.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-21T00:48:21.726 INFO:teuthology.orchestra.run.smithi039.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-21T00:48:21.756 INFO:teuthology.orchestra.run.smithi150.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-21T00:48:21.797 INFO:teuthology.orchestra.run.smithi150.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-21T00:48:21.915 INFO:teuthology.orchestra.run.smithi150.stdout:(150/157): NetworkManager-1.51.0-1.el9.x86_64.r 20 MB/s | 2.3 MB 00:00 2024-09-21T00:48:22.006 INFO:teuthology.orchestra.run.smithi150.stdout:(151/157): NetworkManager-libnm-1.51.0-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-09-21T00:48:22.010 INFO:teuthology.orchestra.run.smithi039.stdout:(150/157): librados2-19.3.0-5120.g18432218.el9. 9.2 MB/s | 3.4 MB 00:00 2024-09-21T00:48:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:(151/157): NetworkManager-1.51.0-1.el9.x86_64.r 7.0 MB/s | 2.3 MB 00:00 2024-09-21T00:48:22.065 INFO:teuthology.orchestra.run.smithi150.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 704 kB/s | 41 kB 00:00 2024-09-21T00:48:22.156 INFO:teuthology.orchestra.run.smithi039.stdout:(152/157): librbd1-19.3.0-5120.g18432218.el9.x8 6.5 MB/s | 3.2 MB 00:00 2024-09-21T00:48:22.165 INFO:teuthology.orchestra.run.smithi150.stdout:(153/157): librados2-19.3.0-5120.g18432218.el9. 7.6 MB/s | 3.4 MB 00:00 2024-09-21T00:48:22.186 INFO:teuthology.orchestra.run.smithi039.stdout:(153/157): NetworkManager-team-1.51.0-1.el9.x86 327 kB/s | 41 kB 00:00 2024-09-21T00:48:22.215 INFO:teuthology.orchestra.run.smithi150.stdout:(154/157): librbd1-19.3.0-5120.g18432218.el9.x8 6.9 MB/s | 3.2 MB 00:00 2024-09-21T00:48:22.249 INFO:teuthology.orchestra.run.smithi150.stdout:(155/157): NetworkManager-tui-1.51.0-1.el9.x86_ 1.3 MB/s | 249 kB 00:00 2024-09-21T00:48:22.269 INFO:teuthology.orchestra.run.smithi039.stdout:(154/157): openssl-3.2.2-6.el9.x86_64.rpm 16 MB/s | 1.4 MB 00:00 2024-09-21T00:48:22.316 INFO:teuthology.orchestra.run.smithi150.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 9.0 MB/s | 1.4 MB 00:00 2024-09-21T00:48:22.370 INFO:teuthology.orchestra.run.smithi039.stdout:(155/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-09-21T00:48:22.395 INFO:teuthology.orchestra.run.smithi039.stdout:(156/157): NetworkManager-tui-1.51.0-1.el9.x86_ 1.0 MB/s | 249 kB 00:00 2024-09-21T00:48:22.437 INFO:teuthology.orchestra.run.smithi039.stdout:(157/157): NetworkManager-libnm-1.51.0-1.el9.x8 4.2 MB/s | 1.8 MB 00:00 2024-09-21T00:48:22.442 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-21T00:48:22.442 INFO:teuthology.orchestra.run.smithi039.stdout:Total 19 MB/s | 228 MB 00:12 2024-09-21T00:48:22.474 INFO:teuthology.orchestra.run.smithi150.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 9.2 MB/s | 2.4 MB 00:00 2024-09-21T00:48:22.479 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-09-21T00:48:22.479 INFO:teuthology.orchestra.run.smithi150.stdout:Total 19 MB/s | 228 MB 00:12 2024-09-21T00:48:23.670 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-09-21T00:48:23.821 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-09-21T00:48:23.821 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-09-21T00:48:24.563 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-21T00:48:24.764 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-21T00:48:24.764 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-21T00:48:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-09-21T00:48:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-09-21T00:48:27.286 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-21T00:48:27.287 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-21T00:48:27.811 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T00:48:27.918 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T00:48:27.956 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T00:48:28.017 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T00:48:28.051 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T00:48:28.077 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T00:48:28.128 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T00:48:28.470 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T00:48:28.578 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:48:28.850 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:48:28.879 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:48:28.971 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:48:29.000 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T00:48:29.030 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T00:48:29.183 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T00:48:29.247 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T00:48:29.275 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:48:29.374 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:48:29.402 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T00:48:29.456 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T00:48:29.477 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T00:48:29.503 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T00:48:29.696 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T00:48:29.837 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T00:48:29.942 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T00:48:30.039 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:30.110 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:30.343 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:30.455 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T00:48:30.481 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T00:48:30.516 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T00:48:30.590 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T00:48:30.634 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T00:48:30.674 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T00:48:30.721 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T00:48:30.753 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T00:48:30.781 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T00:48:30.850 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T00:48:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T00:48:30.950 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T00:48:30.986 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T00:48:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T00:48:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T00:48:31.148 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T00:48:31.177 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T00:48:31.247 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T00:48:31.317 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T00:48:31.390 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T00:48:31.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T00:48:31.577 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T00:48:31.898 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-09-21T00:48:32.020 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T00:48:32.097 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T00:48:32.138 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T00:48:32.180 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T00:48:32.210 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T00:48:32.252 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T00:48:32.280 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T00:48:32.290 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T00:48:32.321 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T00:48:32.332 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T00:48:32.346 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T00:48:32.390 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T00:48:32.476 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T00:48:32.527 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T00:48:32.616 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T00:48:32.690 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T00:48:32.730 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T00:48:32.762 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T00:48:32.795 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T00:48:32.831 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T00:48:32.872 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:48:32.878 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T00:48:32.914 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T00:48:32.956 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T00:48:32.999 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T00:48:32.999 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:48:33.031 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:48:33.122 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:48:33.133 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T00:48:33.152 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T00:48:33.176 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T00:48:33.182 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T00:48:33.216 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T00:48:33.335 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T00:48:33.400 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T00:48:33.428 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:48:33.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T00:48:33.533 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:48:33.550 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T00:48:33.563 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T00:48:33.582 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T00:48:33.608 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T00:48:33.625 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T00:48:33.632 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T00:48:33.647 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T00:48:33.668 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T00:48:33.706 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T00:48:33.857 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T00:48:33.863 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T00:48:33.906 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T00:48:33.933 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T00:48:33.968 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T00:48:33.998 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T00:48:34.007 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T00:48:34.038 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T00:48:34.069 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T00:48:34.103 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T00:48:34.137 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T00:48:34.145 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T00:48:34.178 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T00:48:34.213 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T00:48:34.235 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T00:48:34.316 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T00:48:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T00:48:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T00:48:34.538 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T00:48:34.645 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T00:48:34.659 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-21T00:48:34.679 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T00:48:34.714 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T00:48:34.787 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T00:48:34.809 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T00:48:34.831 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T00:48:34.860 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T00:48:34.871 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T00:48:34.915 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T00:48:34.926 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T00:48:34.956 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T00:48:34.978 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T00:48:34.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T00:48:35.009 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T00:48:35.053 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T00:48:35.096 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T00:48:35.106 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T00:48:35.137 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T00:48:35.172 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T00:48:35.253 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T00:48:35.273 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T00:48:35.283 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T00:48:35.315 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T00:48:35.326 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T00:48:35.349 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T00:48:35.363 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T00:48:35.433 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T00:48:35.446 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T00:48:35.487 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T00:48:35.495 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T00:48:35.501 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T00:48:35.540 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T00:48:35.560 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T00:48:35.618 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T00:48:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:48:35.668 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T00:48:35.694 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T00:48:35.747 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T00:48:35.772 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:48:35.802 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:48:35.852 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T00:48:35.882 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T00:48:35.900 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:48:35.934 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T00:48:35.959 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T00:48:35.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T00:48:36.195 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T00:48:36.257 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T00:48:36.274 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T00:48:36.283 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:48:36.369 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T00:48:36.397 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T00:48:36.413 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:48:36.440 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T00:48:36.450 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T00:48:36.479 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T00:48:36.520 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T00:48:36.533 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T00:48:36.559 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T00:48:36.600 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T00:48:36.605 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T00:48:36.664 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T00:48:36.713 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T00:48:36.723 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T00:48:36.814 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T00:48:36.853 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T00:48:36.871 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T00:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T00:48:36.992 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T00:48:37.028 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T00:48:37.042 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T00:48:37.075 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T00:48:37.128 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T00:48:37.179 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T00:48:37.221 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T00:48:37.258 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T00:48:37.357 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T00:48:37.386 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:37.490 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T00:48:37.493 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:37.615 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T00:48:37.720 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T00:48:37.756 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T00:48:37.805 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T00:48:37.825 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T00:48:37.838 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T00:48:37.863 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T00:48:37.938 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T00:48:37.943 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T00:48:37.990 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T00:48:38.027 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T00:48:38.031 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T00:48:38.055 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T00:48:38.066 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T00:48:38.077 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T00:48:38.088 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T00:48:38.112 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T00:48:38.140 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T00:48:38.182 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T00:48:38.195 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T00:48:38.222 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T00:48:38.253 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T00:48:38.272 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T00:48:38.283 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T00:48:38.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T00:48:38.344 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T00:48:38.351 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T00:48:38.393 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T00:48:38.412 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T00:48:38.423 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T00:48:38.427 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T00:48:38.451 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T00:48:38.486 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T00:48:38.513 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T00:48:38.537 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T00:48:38.542 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T00:48:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T00:48:38.566 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T00:48:38.597 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T00:48:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T00:48:38.644 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T00:48:38.700 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T00:48:38.740 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T00:48:38.747 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T00:48:38.798 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T00:48:38.879 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T00:48:38.993 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T00:48:39.061 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T00:48:39.100 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T00:48:39.129 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T00:48:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T00:48:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T00:48:39.200 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T00:48:39.231 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T00:48:39.284 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T00:48:39.378 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T00:48:39.469 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T00:48:39.505 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T00:48:39.520 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T00:48:39.554 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T00:48:39.651 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T00:48:39.700 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T00:48:39.701 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T00:48:39.745 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T00:48:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T00:48:39.832 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T00:48:39.873 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T00:48:40.056 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T00:48:40.087 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T00:48:40.164 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T00:48:40.488 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T00:48:40.708 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T00:48:40.775 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T00:48:40.801 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T00:48:40.861 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T00:48:40.911 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T00:48:40.932 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T00:48:40.994 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T00:48:41.186 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T00:48:41.244 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T00:48:41.353 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T00:48:41.423 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T00:48:41.461 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:41.543 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T00:48:41.579 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T00:48:41.615 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T00:48:41.674 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T00:48:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T00:48:41.761 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T00:48:41.839 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T00:48:42.018 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T00:48:42.062 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T00:48:42.067 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:42.105 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T00:48:42.115 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T00:48:42.128 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T00:48:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T00:48:42.198 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T00:48:42.198 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T00:48:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:42.226 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T00:48:42.231 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T00:48:42.254 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T00:48:42.268 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T00:48:42.287 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T00:48:42.313 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T00:48:42.368 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T00:48:42.401 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T00:48:42.426 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T00:48:42.547 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T00:48:42.613 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T00:48:42.667 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T00:48:42.725 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T00:48:42.758 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T00:48:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T00:48:42.788 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T00:48:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T00:48:42.832 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T00:48:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T00:48:42.913 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T00:48:42.952 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T00:48:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T00:48:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T00:48:43.057 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T00:48:43.100 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T00:48:43.132 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T00:48:43.253 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T00:48:43.325 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T00:48:43.330 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T00:48:43.361 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T00:48:43.362 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T00:48:43.432 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T00:48:43.463 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T00:48:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T00:48:43.575 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T00:48:44.518 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T00:48:44.577 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T00:48:44.610 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T00:48:44.732 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T00:48:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T00:48:44.847 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T00:48:44.971 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T00:48:45.012 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T00:48:45.259 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T00:48:45.292 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T00:48:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T00:48:45.738 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:45.776 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:46.240 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T00:48:46.343 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:46.387 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T00:48:46.393 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T00:48:46.482 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T00:48:46.483 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T00:48:46.483 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T00:48:46.483 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:48:46.514 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T00:48:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T00:48:48.132 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T00:48:48.168 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T00:48:48.241 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T00:48:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T00:48:48.679 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T00:48:48.785 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T00:48:48.819 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T00:48:48.868 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T00:48:49.467 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T00:48:49.509 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T00:48:49.622 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T00:48:49.651 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T00:48:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:52.988 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T00:48:53.033 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T00:48:53.159 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T00:48:53.159 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T00:48:53.160 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T00:48:53.160 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:48:53.209 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T00:48:54.147 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T00:48:54.147 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2024-09-21T00:48:54.147 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2024-09-21T00:48:54.147 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2024-09-21T00:48:54.147 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2024-09-21T00:48:54.147 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2024-09-21T00:48:54.148 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2024-09-21T00:48:54.148 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2024-09-21T00:48:54.148 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:54.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T00:48:54.343 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T00:48:54.343 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:48:54.343 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T00:48:54.343 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T00:48:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T00:48:55.955 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:48:55.955 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T00:48:55.955 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T00:48:55.955 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T00:48:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:56.187 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T00:48:56.211 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T00:48:56.211 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:48:56.211 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T00:48:56.211 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T00:48:56.211 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T00:48:56.211 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T00:48:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T00:48:56.702 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:48:56.703 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T00:48:56.703 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T00:48:56.703 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T00:48:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:48:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T00:48:56.866 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T00:48:56.921 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T00:48:57.105 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T00:48:57.163 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T00:48:57.240 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T00:48:57.328 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T00:48:57.468 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T00:48:57.504 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T00:48:57.537 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T00:48:57.564 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T00:48:57.991 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T00:48:58.021 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T00:48:58.045 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T00:48:58.103 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T00:48:58.128 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T00:48:58.445 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T00:48:58.445 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-09-21T00:48:58.446 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:48:58.634 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T00:48:58.652 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T00:48:58.652 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:48:58.653 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T00:48:58.653 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:00.262 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T00:49:00.288 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T00:49:00.288 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:00.288 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T00:49:00.288 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T00:49:00.288 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T00:49:00.288 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:00.520 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T00:49:00.544 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T00:49:00.544 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:00.544 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T00:49:00.544 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T00:49:00.544 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T00:49:00.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:01.022 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T00:49:01.045 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T00:49:01.045 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:01.045 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T00:49:01.045 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T00:49:01.045 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T00:49:01.045 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:01.152 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T00:49:01.208 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T00:49:01.264 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T00:49:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T00:49:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T00:49:01.458 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T00:49:01.505 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T00:49:01.575 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T00:49:01.663 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T00:49:01.802 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T00:49:01.847 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T00:49:01.879 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T00:49:01.907 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T00:49:02.341 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T00:49:02.367 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T00:49:02.375 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T00:49:02.405 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T00:49:02.437 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T00:49:02.482 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T00:49:02.503 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T00:49:02.528 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T00:49:02.651 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T00:49:02.677 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T00:49:02.701 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T00:49:02.701 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:02.701 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T00:49:02.702 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T00:49:02.702 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T00:49:02.702 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:02.753 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T00:49:02.765 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T00:49:02.864 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T00:49:02.910 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T00:49:03.879 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T00:49:03.922 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T00:49:03.946 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T00:49:03.946 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:03.946 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T00:49:03.946 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T00:49:03.946 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T00:49:03.946 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:03.990 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T00:49:04.013 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T00:49:04.013 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:04.014 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T00:49:04.014 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:04.065 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T00:49:04.089 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T00:49:04.089 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:04.089 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T00:49:04.089 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T00:49:04.089 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T00:49:04.089 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:04.416 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T00:49:04.440 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T00:49:04.440 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:04.440 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T00:49:04.440 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T00:49:04.440 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T00:49:04.440 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:05.702 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T00:49:05.768 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T00:49:06.602 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T00:49:06.602 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-09-21T00:49:06.602 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-09-21T00:49:06.602 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-09-21T00:49:06.603 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-09-21T00:49:06.603 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-09-21T00:49:06.603 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-09-21T00:49:06.603 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-09-21T00:49:06.603 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:06.729 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T00:49:06.759 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T00:49:06.830 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T00:49:06.872 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T00:49:06.872 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:06.872 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T00:49:06.872 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:06.875 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T00:49:07.042 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T00:49:07.078 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T00:49:07.102 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T00:49:07.102 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:07.103 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T00:49:07.103 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T00:49:07.103 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T00:49:07.103 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:07.154 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T00:49:07.165 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T00:49:07.263 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T00:49:07.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T00:49:08.284 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T00:49:08.315 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T00:49:08.338 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T00:49:08.338 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:08.338 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T00:49:08.338 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T00:49:08.338 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T00:49:08.338 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:08.382 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T00:49:08.403 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T00:49:08.405 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T00:49:08.405 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:08.405 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T00:49:08.405 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:08.444 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T00:49:08.457 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T00:49:08.480 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T00:49:08.481 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:08.481 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T00:49:08.481 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T00:49:08.481 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T00:49:08.481 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:08.484 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T00:49:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T00:49:08.555 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T00:49:08.648 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T00:49:08.707 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T00:49:08.707 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:08.707 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T00:49:08.707 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T00:49:08.707 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T00:49:08.707 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:08.744 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T00:49:08.790 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T00:49:08.800 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T00:49:08.821 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T00:49:08.823 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T00:49:08.823 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:08.823 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T00:49:08.823 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T00:49:08.823 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T00:49:08.823 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:08.850 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T00:49:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T00:49:08.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T00:49:09.025 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T00:49:09.025 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:09.025 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T00:49:09.026 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T00:49:09.026 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T00:49:09.026 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:09.470 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:09.511 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:09.531 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:09.566 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T00:49:09.618 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T00:49:09.618 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:09.618 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T00:49:09.618 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T00:49:09.618 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T00:49:09.619 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:09.751 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T00:49:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T00:49:09.914 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T00:49:10.163 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T00:49:10.254 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T00:49:10.342 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T00:49:10.358 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T00:49:10.449 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T00:49:10.461 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T00:49:10.490 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T00:49:10.570 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T00:49:10.610 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T00:49:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T00:49:10.635 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T00:49:10.661 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T00:49:10.661 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T00:49:10.679 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T00:49:10.718 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T00:49:10.743 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T00:49:10.753 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T00:49:10.754 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T00:49:10.838 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T00:49:10.862 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T00:49:10.868 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:10.868 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:11.039 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:11.039 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T00:49:11.132 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T00:49:11.221 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:11.241 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T00:49:11.276 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T00:49:11.310 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T00:49:11.393 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T00:49:11.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T00:49:12.788 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T00:49:12.829 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T00:49:12.869 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T00:49:12.901 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T00:49:12.928 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T00:49:13.109 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T00:49:13.150 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T00:49:13.181 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T00:49:13.210 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T00:49:13.256 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T00:49:13.876 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:13.912 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:13.932 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:14.767 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T00:49:14.858 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T00:49:14.900 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T00:49:14.979 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T00:49:15.011 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T00:49:15.036 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T00:49:15.062 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T00:49:15.063 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T00:49:15.130 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T00:49:15.130 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T00:49:15.214 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T00:49:15.239 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T00:49:15.244 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:15.244 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:15.290 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T00:49:15.355 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T00:49:15.441 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:15.441 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T00:49:15.525 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T00:49:15.596 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T00:49:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T00:49:16.710 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T00:49:16.900 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T00:49:16.932 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T00:49:16.983 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T00:49:16.983 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:16.983 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T00:49:16.983 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T00:49:16.983 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T00:49:16.983 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T00:49:17.076 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T00:49:17.202 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T00:49:17.276 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T00:49:17.568 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:17.568 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T00:49:17.569 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T00:49:17.570 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T00:49:17.571 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T00:49:17.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T00:49:17.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T00:49:17.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T00:49:17.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T00:49:17.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T00:49:17.585 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T00:49:17.586 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T00:49:17.587 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T00:49:17.588 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T00:49:17.589 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T00:49:17.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T00:49:17.591 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T00:49:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T00:49:18.443 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T00:49:18.495 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T00:49:18.495 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:18.495 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T00:49:18.495 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T00:49:18.496 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T00:49:18.496 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:18.544 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T00:49:18.595 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T00:49:18.595 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:18.595 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T00:49:18.595 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:18.670 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T00:49:18.722 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T00:49:18.722 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:18.722 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T00:49:18.722 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T00:49:18.723 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T00:49:18.723 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:19.104 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T00:49:19.156 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T00:49:19.156 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T00:49:19.156 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T00:49:19.156 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T00:49:19.157 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T00:49:19.157 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:20.126 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.229 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.230 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:20.230 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:20.230 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T00:49:20.230 INFO:teuthology.orchestra.run.smithi160.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T00:49:20.230 INFO:teuthology.orchestra.run.smithi160.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T00:49:20.230 INFO:teuthology.orchestra.run.smithi160.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.231 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.232 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.232 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.232 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.232 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.232 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.232 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.233 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.233 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.233 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.233 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.233 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.233 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T00:49:20.234 INFO:teuthology.orchestra.run.smithi160.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T00:49:20.235 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T00:49:20.236 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T00:49:20.237 INFO:teuthology.orchestra.run.smithi160.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T00:49:20.238 INFO:teuthology.orchestra.run.smithi160.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T00:49:20.239 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T00:49:20.239 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T00:49:20.239 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T00:49:20.239 INFO:teuthology.orchestra.run.smithi160.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T00:49:20.239 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T00:49:20.239 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T00:49:20.240 INFO:teuthology.orchestra.run.smithi160.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T00:49:20.241 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T00:49:20.242 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T00:49:20.243 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T00:49:20.244 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T00:49:20.245 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T00:49:20.246 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T00:49:20.247 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T00:49:20.248 INFO:teuthology.orchestra.run.smithi160.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T00:49:20.248 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:49:20.248 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T00:49:20.567 DEBUG:teuthology.parallel:result is None 2024-09-21T00:49:22.170 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:22.170 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:49:22.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T00:49:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T00:49:22.173 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T00:49:22.174 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T00:49:22.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T00:49:22.176 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T00:49:22.177 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T00:49:22.178 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T00:49:22.179 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T00:49:22.180 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T00:49:22.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T00:49:22.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T00:49:23.768 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T00:49:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T00:49:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T00:49:23.897 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T00:49:23.933 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T00:49:24.210 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T00:49:24.263 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T00:49:24.292 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T00:49:24.324 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T00:49:24.368 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T00:49:24.774 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:24.774 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:24.774 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-09-21T00:49:24.774 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.775 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.776 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T00:49:24.777 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T00:49:24.778 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T00:49:24.779 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T00:49:24.780 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T00:49:24.783 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T00:49:24.784 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T00:49:24.785 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T00:49:24.786 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T00:49:24.786 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T00:49:24.786 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T00:49:24.786 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:49:24.786 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-09-21T00:49:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:25.195 DEBUG:teuthology.parallel:result is None 2024-09-21T00:49:25.225 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T00:49:26.616 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T00:49:26.731 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T00:49:26.801 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T00:49:26.926 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T00:49:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T00:49:26.992 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T00:49:27.028 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T00:49:27.028 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T00:49:27.161 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T00:49:27.161 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T00:49:27.270 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T00:49:27.294 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T00:49:27.310 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:27.311 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:27.563 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T00:49:27.563 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T00:49:27.689 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T00:49:27.767 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:27.985 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:27.985 INFO:teuthology.orchestra.run.smithi039.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-21T00:49:27.985 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T00:49:34.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T00:49:34.271 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T00:49:34.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T00:49:34.273 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T00:49:34.274 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T00:49:34.275 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T00:49:34.276 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T00:49:34.277 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T00:49:34.278 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T00:49:34.279 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T00:49:34.280 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T00:49:34.281 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T00:49:34.282 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T00:49:34.282 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T00:49:34.282 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T00:49:34.282 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T00:49:34.282 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T00:49:36.682 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.683 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.684 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T00:49:36.685 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T00:49:36.686 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T00:49:36.687 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T00:49:36.688 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T00:49:36.689 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T00:49:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T00:49:36.691 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T00:49:36.692 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T00:49:36.693 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T00:49:36.694 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T00:49:36.694 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:49:36.694 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-21T00:49:37.527 DEBUG:teuthology.parallel:result is None 2024-09-21T00:49:37.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:49:37.665 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T00:49:37.717 INFO:teuthology.orchestra.run.smithi039.stdout:19.3.0-5120.g18432218.el9 2024-09-21T00:49:37.718 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T00:49:37.718 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T00:49:37.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:49:37.859 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T00:49:37.886 INFO:teuthology.orchestra.run.smithi150.stdout:19.3.0-5120.g18432218.el9 2024-09-21T00:49:37.886 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T00:49:37.886 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T00:49:37.888 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:49:38.043 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T00:49:38.070 INFO:teuthology.orchestra.run.smithi160.stdout:19.3.0-5120.g18432218.el9 2024-09-21T00:49:38.070 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T00:49:38.070 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T00:49:38.072 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T00:49:38.072 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:49:38.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T00:49:38.136 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:49:38.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T00:49:38.169 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:49:38.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T00:49:38.201 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T00:49:38.201 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:49:38.201 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T00:49:38.260 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T00:49:38.359 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:49:38.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T00:49:38.385 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T00:49:38.451 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:49:38.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T00:49:38.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T00:49:38.550 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T00:49:38.550 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:49:38.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T00:49:38.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T00:49:38.710 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:49:38.711 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T00:49:38.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T00:49:38.800 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:49:38.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T00:49:38.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T00:49:38.891 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T00:49:38.891 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:49:38.891 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T00:49:38.953 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T00:49:39.052 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:49:39.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T00:49:39.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T00:49:39.141 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:49:39.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T00:49:39.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T00:49:39.232 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T00:49:39.328 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug', 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'mgr-modules': ['snap_schedule'], 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'subvols': {'create': 2}} 2024-09-21T00:49:39.328 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:49:39.328 INFO:tasks.cephadm:Cluster fsid is 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:49:39.328 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T00:49:39.329 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.39', 'mon.b': '172.21.15.150', 'mon.c': '172.21.15.160'} 2024-09-21T00:49:39.329 INFO:tasks.cephadm:First mon is mon.a on smithi039 2024-09-21T00:49:39.329 INFO:tasks.cephadm:First mgr is x 2024-09-21T00:49:39.329 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T00:49:39.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname $(hostname -s) 2024-09-21T00:49:39.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-09-21T00:49:39.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-09-21T00:49:39.437 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T00:49:39.438 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:49:39.575 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 00:04:17.484514', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 22:54:16.499090', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T00:49:39.715 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20240920.212106-debug, sha1 1843221895adccfb3b692060251dd6a0183bc410 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:49:39.717 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T00:49:39.717 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T00:49:39.717 DEBUG:teuthology.orchestra.run.smithi039:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:40.563 INFO:teuthology.orchestra.run.smithi039.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 00:49 /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:40.563 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:41.445 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 00:49 /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:41.445 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:42.159 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 00:49 /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:42.159 DEBUG:teuthology.orchestra.run.smithi039:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:42.201 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:42.221 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T00:49:42.254 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 on all hosts... 2024-09-21T00:49:42.254 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T00:49:42.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T00:49:42.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T00:49:43.809 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T00:49:44.154 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T00:49:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T00:50:16.618 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T00:50:16.618 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T00:50:16.618 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T00:50:16.618 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-09-21T00:50:16.618 INFO:teuthology.orchestra.run.smithi160.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T00:50:16.619 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T00:50:16.619 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-09-21T00:50:36.972 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-21T00:50:42.149 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T00:50:42.150 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T00:50:42.150 INFO:teuthology.orchestra.run.smithi039.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T00:50:42.150 INFO:teuthology.orchestra.run.smithi039.stdout: "repo_digests": [ 2024-09-21T00:50:42.150 INFO:teuthology.orchestra.run.smithi039.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T00:50:42.150 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T00:50:42.150 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T00:50:42.182 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph 2024-09-21T00:50:42.249 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-09-21T00:50:42.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-09-21T00:50:42.316 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /etc/ceph 2024-09-21T00:50:42.372 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-09-21T00:50:42.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-09-21T00:50:42.431 INFO:tasks.cephadm:Writing seed config... 2024-09-21T00:50:42.433 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-21T00:50:42.433 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-21T00:50:42.433 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T00:50:42.433 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T00:50:42.433 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T00:50:42.434 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T00:50:42.434 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T00:50:42.434 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-21T00:50:42.434 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-21T00:50:42.434 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-21T00:50:42.434 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-21T00:50:42.435 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-21T00:50:42.436 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-21T00:50:42.436 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-21T00:50:42.436 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T00:50:42.436 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-09-21T00:50:42.437 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:50:42.437 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T00:50:42.472 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 61abfb08-77b3-11ef-baf5-efdc52797490 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T00:50:42.473 DEBUG:teuthology.orchestra.run.smithi039:mon.a> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service 2024-09-21T00:50:42.517 DEBUG:teuthology.orchestra.run.smithi039:mgr.x> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x.service 2024-09-21T00:50:42.562 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T00:50:42.562 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 -v bootstrap --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.39 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T00:50:42.825 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-21T00:50:42.825 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', '-v', 'bootstrap', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.39', '--skip-admin-label'] 2024-09-21T00:50:42.825 INFO:teuthology.orchestra.run.smithi039.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T00:50:42.825 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying podman|docker is present... 2024-09-21T00:50:42.871 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 5.2.2 2024-09-21T00:50:42.871 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying lvm2 is present... 2024-09-21T00:50:42.871 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying time synchronization is in place... 2024-09-21T00:50:42.885 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T00:50:42.886 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T00:50:42.898 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T00:50:42.898 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-09-21T00:50:42.912 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2024-09-21T00:50:42.924 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2024-09-21T00:50:42.924 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2024-09-21T00:50:42.924 INFO:teuthology.orchestra.run.smithi039.stdout:Repeating the final host check... 2024-09-21T00:50:42.969 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 5.2.2 2024-09-21T00:50:42.969 INFO:teuthology.orchestra.run.smithi039.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T00:50:42.969 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl is present 2024-09-21T00:50:42.969 INFO:teuthology.orchestra.run.smithi039.stdout:lvcreate is present 2024-09-21T00:50:42.982 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T00:50:42.982 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T00:50:42.994 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T00:50:42.994 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-09-21T00:50:43.008 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2024-09-21T00:50:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2024-09-21T00:50:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2024-09-21T00:50:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:Host looks OK 2024-09-21T00:50:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:Cluster fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:50:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:Acquiring lock 140190924929680 on /run/cephadm/61abfb08-77b3-11ef-baf5-efdc52797490.lock 2024-09-21T00:50:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:Lock 140190924929680 acquired on /run/cephadm/61abfb08-77b3-11ef-baf5-efdc52797490.lock 2024-09-21T00:50:43.022 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 3300 ... 2024-09-21T00:50:43.022 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 6789 ... 2024-09-21T00:50:43.022 INFO:teuthology.orchestra.run.smithi039.stdout:Base mon IP(s) is [172.21.15.39:3300, 172.21.15.39:6789], mon addrv is [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-09-21T00:50:43.029 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.39 metric 100 2024-09-21T00:50:43.029 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.39 metric 100 2024-09-21T00:50:43.035 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T00:50:43.035 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-21T00:50:43.035 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1742sec hoplimit 64 pref medium 2024-09-21T00:50:43.040 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T00:50:43.040 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T00:50:43.040 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T00:50:43.041 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-21T00:50:43.041 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1370/64 scope link 2024-09-21T00:50:43.041 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T00:50:43.041 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2024-09-21T00:50:43.041 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2024-09-21T00:50:43.041 INFO:teuthology.orchestra.run.smithi039.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T00:50:43.042 INFO:teuthology.orchestra.run.smithi039.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T00:50:43.043 INFO:teuthology.orchestra.run.smithi039.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T00:50:43.611 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T00:50:43.611 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T00:50:43.611 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T00:50:43.611 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T00:50:43.611 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:8425bc7408d91b180e7fdc62f23e7ae6603757f4f36e9969a4eaf9fd9416b764 2024-09-21T00:50:43.611 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying config sha256:ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T00:50:43.612 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T00:50:44.716 INFO:teuthology.orchestra.run.smithi039.stdout:ceph: stdout ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T00:50:44.716 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph version: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T00:50:44.716 INFO:teuthology.orchestra.run.smithi039.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T00:50:45.950 INFO:teuthology.orchestra.run.smithi039.stdout:stat: stdout 167 167 2024-09-21T00:50:45.950 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial keys... 2024-09-21T00:50:47.003 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQBmGO5mDkSTGRAA8Vrk4zhI1RDa5Q9qRhHhcQ== 2024-09-21T00:50:48.291 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQBnGO5mayi1JxAAFWEcdj+L5kYcyfG9hJs0kA== 2024-09-21T00:50:49.375 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQBoGO5muVeFLhAAPNs5P8N/hMkmsp49MGWL5A== 2024-09-21T00:50:49.376 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial monmap... 2024-09-21T00:50:50.442 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool for a [v2:172.21.15.39:3300,v1:172.21.15.39:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = quincy 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: set fsid to 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:50:50.443 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mon... 2024-09-21T00:50:51.630 INFO:teuthology.orchestra.run.smithi039.stdout:create mon.a on 2024-09-21T00:50:51.951 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T00:50:52.179 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T00:50:52.446 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-61abfb08-77b3-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-61abfb08-77b3-11ef-baf5-efdc52797490.target. 2024-09-21T00:50:52.446 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-61abfb08-77b3-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-61abfb08-77b3-11ef-baf5-efdc52797490.target. 2024-09-21T00:50:55.824 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a 2024-09-21T00:50:55.824 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-21T00:50:56.340 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-61abfb08-77b3-11ef-baf5-efdc52797490.target.wants/ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-61abfb08-77b3-11ef-baf5-efdc52797490@.service. 2024-09-21T00:50:57.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:50:56 smithi039 podman[30025]: 2024-09-21 00:50:56.591630094 +0000 UTC m=+0.027448627 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:51:01.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:00 smithi039 podman[30025]: 2024-09-21 00:51:00.905648682 +0000 UTC m=+4.341467197 container create 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T00:51:02.280 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T00:51:02.280 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-09-21T00:51:02.295 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T00:51:02.295 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-09-21T00:51:02.295 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-09-21T00:51:02.295 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T00:51:02.296 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon to start... 2024-09-21T00:51:02.296 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon... 2024-09-21T00:51:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:02 smithi039 podman[30025]: 2024-09-21 00:51:02.162461613 +0000 UTC m=+5.598280131 container init 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T00:51:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:02 smithi039 podman[30025]: 2024-09-21 00:51:02.171286949 +0000 UTC m=+5.607105467 container start 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T00:51:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:02 smithi039 bash[30025]: 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e 2024-09-21T00:51:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:02 smithi039 systemd[1]: Started Ceph mon.a for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:51:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:02 smithi039 ceph-mon[30052]: mkfs 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:02.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:02 smithi039 ceph-mon[30052]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T00:51:03.656 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T00:51:03.658 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: monmap epoch 1 2024-09-21T00:51:03.658 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:03.658 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: last_changed 2024-09-21T00:50:49.850177+0000 2024-09-21T00:51:03.658 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: min_mon_release 19 (squid) 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: election_strategy: 1 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: fsmap 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: mgrmap e1: no daemons active 2024-09-21T00:51:03.659 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:03 smithi039 ceph-mon[30052]: from='client.? 172.21.15.39:0/26083842' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout id: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout services: 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.644404s) 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T00:51:03.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:03.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout data: 2024-09-21T00:51:03.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T00:51:03.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T00:51:03.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T00:51:03.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T00:51:03.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:03.871 INFO:teuthology.orchestra.run.smithi039.stdout:mon is available 2024-09-21T00:51:03.872 INFO:teuthology.orchestra.run.smithi039.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T00:51:05.099 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:04 smithi039 ceph-mon[30052]: from='client.? 172.21.15.39:0/2187655056' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T00:51:05.099 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:04 smithi039 ceph-mon[30052]: from='client.? 172.21.15.39:0/2187655056' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T00:51:05.367 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:05.367 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T00:51:05.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T00:51:05.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T00:51:05.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T00:51:05.369 INFO:teuthology.orchestra.run.smithi039.stdout:Generating new minimal ceph.conf... 2024-09-21T00:51:06.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:06 smithi039 ceph-mon[30052]: from='client.? 172.21.15.39:0/964853525' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:51:06.815 INFO:teuthology.orchestra.run.smithi039.stdout:Restarting the monitor... 2024-09-21T00:51:07.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:06 smithi039 systemd[1]: Stopping Ceph mon.a for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:51:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a[30040]: 2024-09-21T00:51:07.242+0000 7f855c600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T00:51:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a[30040]: 2024-09-21T00:51:07.242+0000 7f855c600640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T00:51:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:07 smithi039 podman[30294]: 2024-09-21 00:51:07.469031022 +0000 UTC m=+0.393175636 container died 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T00:51:08.108 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:07 smithi039 podman[30294]: 2024-09-21 00:51:07.906382105 +0000 UTC m=+0.830526721 container cleanup 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True) 2024-09-21T00:51:08.108 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:07 smithi039 bash[30294]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a 2024-09-21T00:51:08.371 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 podman[30306]: 2024-09-21 00:51:08.107392355 +0000 UTC m=+0.631936972 container remove 36504de138bb0847e0b33a36c7bc57b4c1c36b175853c4c72c5e94a2b821953e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T00:51:08.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T00:51:08.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 systemd[1]: Stopped Ceph mon.a for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:51:08.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service: Consumed 1.355s CPU time. 2024-09-21T00:51:08.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 systemd[1]: Starting Ceph mon.a for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:51:08.952 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 podman[30396]: 2024-09-21 00:51:08.605206429 +0000 UTC m=+0.027443133 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:51:08.952 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 podman[30396]: 2024-09-21 00:51:08.708158956 +0000 UTC m=+0.130395648 container create 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T00:51:08.952 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 podman[30396]: 2024-09-21 00:51:08.94412733 +0000 UTC m=+0.366364022 container init 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T00:51:08.952 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:08 smithi039 podman[30396]: 2024-09-21 00:51:08.951013731 +0000 UTC m=+0.373250429 container start 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T00:51:09.052 INFO:teuthology.orchestra.run.smithi039.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 bash[30396]: 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 systemd[1]: Started Ceph mon.a for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: pidfile_write: ignore empty --pid-file 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: load: jerasure load: lrc 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: RocksDB version: 7.9.2 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Git sha 0 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T00:51:09.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: DB SUMMARY 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: DB Session ID: O9BTWY5DVYOABV500SED 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: CURRENT file: CURRENT 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 93704 ; 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.error_if_exists: 0 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.create_if_missing: 0 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.paranoid_checks: 1 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.env: 0x5591a828bd60 2024-09-21T00:51:09.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.info_log: 0x5591e15ad320 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.statistics: (nil) 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.use_fsync: 0 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_log_file_size: 0 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.allow_fallocate: 1 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T00:51:09.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.use_direct_reads: 0 2024-09-21T00:51:09.313 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T00:51:09.313 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T00:51:09.313 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.db_log_dir: 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.wal_dir: 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T00:51:09.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.write_buffer_manager: 0x5591e15b1d60 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.unordered_write: 0 2024-09-21T00:51:09.315 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.row_cache: None 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.wal_filter: None 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.two_write_queues: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.wal_compression: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.atomic_flush: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T00:51:09.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.log_readahead_size: 0 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_background_jobs: 2 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_background_compactions: -1 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_subcompactions: 1 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T00:51:09.317 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_open_files: -1 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_background_flushes: -1 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Compression algorithms supported: 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kZSTD supported: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kXpressCompression supported: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kBZip2Compression supported: 0 2024-09-21T00:51:09.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kLZ4Compression supported: 1 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kZlibCompression supported: 1 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: kSnappyCompression supported: 1 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.merge_operator: 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_filter: None 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T00:51:09.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5591e15acf60) 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: cache_index_and_filter_blocks: 1 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: pin_top_level_index_and_filter: 1 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_type: 0 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: data_block_index_type: 0 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_shortening: 1 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: checksum: 4 2024-09-21T00:51:09.320 INFO:journalctl@ceph.mon.a.smithi039.stdout: no_block_cache: 0 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache: 0x5591dd571610 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_name: BinnedLRUCache 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_options: 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: capacity : 536870912 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: num_shard_bits : 4 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: strict_capacity_limit : 0 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: high_pri_pool_ratio: 0.000 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_compressed: (nil) 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: persistent_cache: (nil) 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_size: 4096 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_size_deviation: 10 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_restart_interval: 16 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_block_restart_interval: 1 2024-09-21T00:51:09.321 INFO:journalctl@ceph.mon.a.smithi039.stdout: metadata_block_size: 4096 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: partition_filters: 0 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: use_delta_encoding: 1 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: filter_policy: bloomfilter 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: whole_key_filtering: 1 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: verify_compression: 0 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: read_amp_bytes_per_bit: 0 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: format_version: 5 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: enable_index_compression: 1 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_align: 0 2024-09-21T00:51:09.322 INFO:journalctl@ceph.mon.a.smithi039.stdout: max_auto_readahead_size: 262144 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout: prepopulate_block_cache: 0 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout: initial_auto_readahead_size: 8192 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression: NoCompression 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.num_levels: 7 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T00:51:09.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T00:51:09.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T00:51:09.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T00:51:09.326 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.inplace_update_support: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.bloom_locality: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.max_successive_merges: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T00:51:09.327 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.ttl: 2592000 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.enable_blob_files: false 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.min_blob_size: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T00:51:09.328 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a8ef4691-b192-457d-88e6-e6936a8bbc12 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726879869062685, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726879869065660, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 89973, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 279, "table_properties": {"data_size": 88043, "index_size": 250, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11333, "raw_average_key_size": 47, "raw_value_size": 81648, "raw_average_value_size": 343, "num_data_blocks": 11, "num_entries": 238, "num_filter_entries": 238, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726879869, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a8ef4691-b192-457d-88e6-e6936a8bbc12", "db_session_id": "O9BTWY5DVYOABV500SED", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726879869065891, "job": 1, "event": "recovery_finished"} 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5591e15d2a80 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: DB pointer 0x5591e2374000 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T00:51:09.329 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** DB Stats ** 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: L0 2/0 89.72 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 48.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Sum 2/0 89.72 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 48.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T00:51:09.330 INFO:journalctl@ceph.mon.a.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 48.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 48.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T00:51:09.331 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative compaction: 0.00 GB write, 8.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval compaction: 0.00 GB write, 8.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: Block cache BinnedLRUCache@0x5591dd571610#2 capacity: 512.00 MB usage: 1.28 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: starting mon.a rank 0 at public addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] at bind addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???) e1 preinit fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).mds e1 new map 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).mds e1 print_map 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: e1 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: btime 2024-09-21T00:51:02:310817+0000 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T00:51:09.332 INFO:journalctl@ceph.mon.a.smithi039.stdout: legacy client fscid: -1 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout: No filesystems configured 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: monmap epoch 1 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: last_changed 2024-09-21T00:50:49.850177+0000 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: min_mon_release 19 (squid) 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: election_strategy: 1 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:51:09.333 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: fsmap 2024-09-21T00:51:09.334 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T00:51:09.334 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:09 smithi039 ceph-mon[30415]: mgrmap e1: no daemons active 2024-09-21T00:51:10.394 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T00:51:10.417 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T00:51:10.417 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mgr... 2024-09-21T00:51:10.418 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T00:51:10.418 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T00:51:10.775 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x 2024-09-21T00:51:10.775 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x.service: Unit ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x.service not loaded. 2024-09-21T00:51:11.012 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-61abfb08-77b3-11ef-baf5-efdc52797490.target.wants/ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x.service → /etc/systemd/system/ceph-61abfb08-77b3-11ef-baf5-efdc52797490@.service. 2024-09-21T00:51:11.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:10 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3437993817' entity='client.admin' 2024-09-21T00:51:11.349 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 systemd[1]: Starting Ceph mgr.x for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:51:11.653 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 podman[30623]: 2024-09-21 00:51:11.253347733 +0000 UTC m=+0.027169750 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T00:51:11.653 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 podman[30623]: 2024-09-21 00:51:11.462228426 +0000 UTC m=+0.236050458 container create 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-21T00:51:11.777 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T00:51:11.778 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-09-21T00:51:11.791 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T00:51:11.791 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-09-21T00:51:11.791 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-09-21T00:51:11.791 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T00:51:11.808 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T00:51:11.809 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-09-21T00:51:11.822 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T00:51:11.822 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-09-21T00:51:11.822 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-09-21T00:51:11.822 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T00:51:11.823 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr to start... 2024-09-21T00:51:11.823 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr... 2024-09-21T00:51:11.964 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 podman[30623]: 2024-09-21 00:51:11.656424865 +0000 UTC m=+0.430246880 container init 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:51:11.964 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 podman[30623]: 2024-09-21 00:51:11.662837879 +0000 UTC m=+0.436659871 container start 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T00:51:11.964 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 bash[30623]: 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 2024-09-21T00:51:11.964 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:11 smithi039 systemd[1]: Started Ceph mgr.x for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:51:12.873 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:12 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2939923360' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.197 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T00:51:13.198 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:13.199 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T00:51:13.199 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T00:51:13.199 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T00:51:13.199 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T00:51:13.199 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T00:51:13.199 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T00:51:13.202 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T00:51:02:310817+0000", 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T00:51:13.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T00:51:02.312000+0000", 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:51:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (1/15)... 2024-09-21T00:51:16.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:15 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2868497058' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T00:51:16.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:16.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:51:16.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T00:51:16.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T00:51:16.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T00:51:16.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T00:51:16.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T00:51:16.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T00:51:16.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T00:51:16.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-09-21T00:51:16.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:16.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T00:51:16.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T00:51:16.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:16.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-21T00:51:16.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T00:51:16.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.555 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T00:51:02:310817+0000", 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T00:51:16.556 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T00:51:02.312000+0000", 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:51:16.557 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (2/15)... 2024-09-21T00:51:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: Activating manager daemon x 2024-09-21T00:51:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: mgrmap e2: x(active, starting, since 0.00483962s) 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: Manager daemon x is now available 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' 2024-09-21T00:51:18.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:18 smithi039 ceph-mon[30415]: from='mgr.14100 172.21.15.39:0/1960750929' entity='mgr.x' 2024-09-21T00:51:20.121 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:20.121 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:51:20.121 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T00:51:20.121 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T00:51:20.122 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T00:51:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T00:51:20.126 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T00:51:20.126 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T00:51:20.126 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T00:51:20.126 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T00:51:02:310817+0000", 2024-09-21T00:51:20.127 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T00:51:20.128 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T00:51:02.312000+0000", 2024-09-21T00:51:20.129 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T00:51:20.129 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-09-21T00:51:20.129 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T00:51:20.129 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:51:20.129 INFO:teuthology.orchestra.run.smithi039.stdout:mgr is available 2024-09-21T00:51:20.363 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:20 smithi039 ceph-mon[30415]: mgrmap e3: x(active, since 1.00936s) 2024-09-21T00:51:20.363 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:20 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2009804440' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T00:51:21.308 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:21 smithi039 ceph-mon[30415]: mgrmap e4: x(active, since 2s) 2024-09-21T00:51:21.308 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:21 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2369209825' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T00:51:21.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:21.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T00:51:21.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T00:51:21.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T00:51:21.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T00:51:21.651 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling cephadm module... 2024-09-21T00:51:22.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:22 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3886157724' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T00:51:22.903 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:22 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ignoring --setuser ceph since I am not root 2024-09-21T00:51:22.903 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:22 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ignoring --setgroup ceph since I am not root 2024-09-21T00:51:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:23 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3886157724' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T00:51:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:23 smithi039 ceph-mon[30415]: mgrmap e5: x(active, since 4s) 2024-09-21T00:51:24.781 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:24 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2445055749' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2024-09-21T00:51:24.791 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 5... 2024-09-21T00:51:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: Active manager daemon x restarted 2024-09-21T00:51:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: Activating manager daemon x 2024-09-21T00:51:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: mgrmap e6: x(active, starting, since 0.00443959s) 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T00:51:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:28 smithi039 ceph-mon[30415]: Manager daemon x is now available 2024-09-21T00:51:30.540 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:30.540 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:30.540 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: Found migration_current of "None". Setting to last migration. 2024-09-21T00:51:30.540 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:30.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:30.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:51:30.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:51:30.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T00:51:30.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T00:51:30.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-mon[30415]: mgrmap e7: x(active, since 1.00734s) 2024-09-21T00:51:30.546 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:51:30.546 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T00:51:30.546 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T00:51:30.546 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:51:30.546 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 5 is available 2024-09-21T00:51:30.547 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying orchestrator module is enabled... 2024-09-21T00:51:31.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T00:51:30.911+0000 7fa3ede00640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:00:51:30] ENGINE Error in HTTPServer.serve 2024-09-21T00:51:31.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: Traceback (most recent call last): 2024-09-21T00:51:31.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:51:31.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._connections.run(self.expiration_interval) 2024-09-21T00:51:31.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:51:31.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._run(expiration_interval) 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: s = self.context.wrap_socket( 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: return self.sslsocket_class._create( 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self.do_handshake() 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._sslobj.do_handshake() 2024-09-21T00:51:31.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:51:31.156 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T00:51:31.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:31 smithi039 ceph-mon[30415]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T00:51:31.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:31 smithi039 ceph-mon[30415]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T00:51:31.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:31 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:51:31.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:31 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2626792116' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T00:51:32.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T00:51:32.068 INFO:teuthology.orchestra.run.smithi039.stdout:Setting orchestrator backend to cephadm... 2024-09-21T00:51:32.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: [21/Sep/2024:00:51:30] ENGINE Bus STARTING 2024-09-21T00:51:32.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: [21/Sep/2024:00:51:30] ENGINE Serving on https://172.21.15.39:7150 2024-09-21T00:51:32.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: [21/Sep/2024:00:51:30] ENGINE Error in HTTPServer.serve 2024-09-21T00:51:32.608 INFO:journalctl@ceph.mon.a.smithi039.stdout: Traceback (most recent call last): 2024-09-21T00:51:32.608 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:51:32.608 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._connections.run(self.expiration_interval) 2024-09-21T00:51:32.608 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:51:32.608 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._run(expiration_interval) 2024-09-21T00:51:32.608 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:51:32.609 INFO:journalctl@ceph.mon.a.smithi039.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:51:32.609 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:51:32.609 INFO:journalctl@ceph.mon.a.smithi039.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:51:32.609 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:51:32.609 INFO:journalctl@ceph.mon.a.smithi039.stdout: s = self.context.wrap_socket( 2024-09-21T00:51:32.609 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:51:32.610 INFO:journalctl@ceph.mon.a.smithi039.stdout: return self.sslsocket_class._create( 2024-09-21T00:51:32.610 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:51:32.610 INFO:journalctl@ceph.mon.a.smithi039.stdout: self.do_handshake() 2024-09-21T00:51:32.610 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:51:32.610 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._sslobj.do_handshake() 2024-09-21T00:51:32.610 INFO:journalctl@ceph.mon.a.smithi039.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:51:32.611 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: [21/Sep/2024:00:51:31] ENGINE Serving on http://172.21.15.39:8765 2024-09-21T00:51:32.611 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: [21/Sep/2024:00:51:31] ENGINE Bus STARTED 2024-09-21T00:51:32.611 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2626792116' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T00:51:32.611 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:32 smithi039 ceph-mon[30415]: mgrmap e8: x(active, since 2s) 2024-09-21T00:51:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:33 smithi039 ceph-mon[30415]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:33 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:33 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:51:35.041 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:34 smithi039 ceph-mon[30415]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:35.046 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T00:51:35.046 INFO:teuthology.orchestra.run.smithi039.stdout:Generating ssh key... 2024-09-21T00:51:36.153 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: Generating public/private rsa key pair. 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: Your identification has been saved in /tmp/tmp0ddrl1_s/key 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: Your public key has been saved in /tmp/tmp0ddrl1_s/key.pub 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: The key fingerprint is: 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: SHA256:U5eTCT/r6yKCvFzRMLS/g5Cm+SxcoOtctgcD1Vb+Res ceph-61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: The key's randomart image is: 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: +---[RSA 3072]----+ 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: | . o. .. | 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: | . +.. .o.+ | 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: | . . +. .oO | 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: | .. . =..o. + | 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: | ...+ . S. E. | 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: |. *.. o o . | 2024-09-21T00:51:36.154 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: | oo=o.o o . | 2024-09-21T00:51:36.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: |o =++o. ... . | 2024-09-21T00:51:36.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: |.o o*. . . oo | 2024-09-21T00:51:36.155 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: +----[SHA256]-----+ 2024-09-21T00:51:37.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:36 smithi039 ceph-mon[30415]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:37.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:36 smithi039 ceph-mon[30415]: Generating ssh key... 2024-09-21T00:51:37.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:36 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:37.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:36 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:37.767 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:51:37.767 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T00:51:37.767 INFO:teuthology.orchestra.run.smithi039.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T00:51:37.772 INFO:teuthology.orchestra.run.smithi039.stdout:Adding host smithi039... 2024-09-21T00:51:38.071 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:37 smithi039 ceph-mon[30415]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:39.041 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:38 smithi039 ceph-mon[30415]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:41.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:40 smithi039 ceph-mon[30415]: Deploying cephadm binary to smithi039 2024-09-21T00:51:43.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Added host 'smithi039' with addr '172.21.15.39' 2024-09-21T00:51:43.649 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying unmanaged mon service... 2024-09-21T00:51:44.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:43 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:44.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:43 smithi039 ceph-mon[30415]: Added host smithi039 2024-09-21T00:51:44.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:43 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:51:45.203 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T00:51:45.204 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying unmanaged mgr service... 2024-09-21T00:51:45.689 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:45 smithi039 ceph-mon[30415]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:45.689 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:45 smithi039 ceph-mon[30415]: Saving service mon spec with placement count:5 2024-09-21T00:51:45.689 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:45 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:47.395 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T00:51:47.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:47 smithi039 ceph-mon[30415]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:47.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:47 smithi039 ceph-mon[30415]: Saving service mgr spec with placement count:2 2024-09-21T00:51:47.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:47 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:47.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:47 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:47.607 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:47 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:49.542 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:49 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2153662753' entity='client.admin' 2024-09-21T00:51:50.577 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:50 smithi039 ceph-mon[30415]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:51:50.577 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:50 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:50.577 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:50 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1744331497' entity='client.admin' 2024-09-21T00:51:50.860 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling the dashboard module... 2024-09-21T00:51:52.385 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:51 smithi039 ceph-mon[30415]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:51:52.385 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:51 smithi039 ceph-mon[30415]: from='mgr.14118 172.21.15.39:0/2303138398' entity='mgr.x' 2024-09-21T00:51:52.385 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:51 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3849219384' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T00:51:53.077 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ignoring --setuser ceph since I am not root 2024-09-21T00:51:53.078 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:51:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ignoring --setgroup ceph since I am not root 2024-09-21T00:51:53.626 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:53 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3849219384' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T00:51:53.626 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:53 smithi039 ceph-mon[30415]: mgrmap e9: x(active, since 23s) 2024-09-21T00:51:54.807 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:54 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3833906155' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2024-09-21T00:51:54.813 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 9... 2024-09-21T00:51:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: Active manager daemon x restarted 2024-09-21T00:51:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: Activating manager daemon x 2024-09-21T00:51:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T00:51:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: mgrmap e10: x(active, starting, since 0.00626426s) 2024-09-21T00:51:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:51:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T00:51:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T00:51:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T00:51:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T00:51:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:58 smithi039 ceph-mon[30415]: Manager daemon x is now available 2024-09-21T00:52:00.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:59 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:00.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:59 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T00:52:00.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:59 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T00:52:00.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:51:59 smithi039 ceph-mon[30415]: mgrmap e11: x(active, since 1.01152s) 2024-09-21T00:52:00.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-09-21T00:52:00.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T00:52:00.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T00:52:00.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-09-21T00:52:00.612 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 9 is available 2024-09-21T00:52:00.612 INFO:teuthology.orchestra.run.smithi039.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T00:52:01.378 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:00 smithi039 ceph-mon[30415]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T00:52:01.378 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:00 smithi039 ceph-mon[30415]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T00:52:01.987 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T00:52:01.720+0000 7fdecbe00640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:00:52:01] ENGINE Error in HTTPServer.serve 2024-09-21T00:52:01.987 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: Traceback (most recent call last): 2024-09-21T00:52:01.987 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:52:01.987 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._connections.run(self.expiration_interval) 2024-09-21T00:52:01.987 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._run(expiration_interval) 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: s = self.context.wrap_socket( 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: return self.sslsocket_class._create( 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self.do_handshake() 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:52:01.988 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._sslobj.do_handshake() 2024-09-21T00:52:01.989 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:52:01.989 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T00:52:02.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-mon[30415]: mgrmap e12: x(active, since 2s) 2024-09-21T00:52:02.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-mon[30415]: [21/Sep/2024:00:52:01] ENGINE Bus STARTING 2024-09-21T00:52:02.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-mon[30415]: [21/Sep/2024:00:52:01] ENGINE Serving on http://172.21.15.39:8765 2024-09-21T00:52:02.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:01 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:03.094 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUB/wFX0j3Clmb+i9D4/aSkKzaefQwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTAwNTIwMloXDTM0\nMDkyMjAwNTIwMlowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjM5MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAvflZfI+A6zDuQy2f2CGfgk8p5aDnzyIx58Fp\nZLKnWhq2GaMXIlUWBhSNUlwbKJQaeRpAkjubqHJd7HqU6TXkONDd47UfMcbtg8hQ\nMDuw9kT1vpZWwdwYbeE9P5jQW9XqBS7dTB7/sla9S3g3UiNvyPDXQL2jU3dxW/4B\nOI71B6IYi0Ecx2HRXZZ67XRiJpjNYFBkzGC5VameFDd1rFwWlLy8zsWOyZP4L/Wd\nIyvFlTToNdPB9WaSc5m6UlGdyRRIcNp5ZOw6oP08Aax7FV9ObIVeVWgD6QsRbLDG\nvZvZztxP/UESRTj7OnUlGRxpKpMdctVhnw4bQdb4WFZquCfu22O5c3b6itz0Nur2\nZfZ+1T6qB7jh1jxRK8fjGxkymaE2dVY84LX4qACzXyZi/Hm6zz+A6HhcPfqqSqH8\nDDSGKJ8mh921r3R3hpdKCe7/XeC14Fp7/upT/vsCcRM2VkcIKMHJRFAhQJsuz5SK\n/WWRsufqjXWJH4bwcou3IlbjafW2BK1IkCWgXAzbElMMMDkrFJ1iWhiYF3GuUWrZ\nJTuk3BNuDCsgv9oHUpXYiGxEmg/tZ9wgGT38GDWb4ObaerVRldbBc2kfMkVlt8jA\nW4AcoAlePCpw3GKQoISkzYNzeJ1JqOq3l16P2mYowO1bQpdHgNa3comrwHUzG3nG\nAPnOJX0CAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDM5ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPJzAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAK\nZVY7ysdQyGJfbp+CVSc8LA6yIWmkVkaLJGXnrnsEPIwMMCDGc7w1DG34dZDQDMdH\nLbNXzKqdn6fiwLMGGIvuXgqRA4ilhmbBlqo5FFTP6fu5hCzs6Loaf7zHEXTmFCDM\nfzSkkDgL2jqU1VTFyv67wNRfsMB4DHuoVvQEsjBVs2kfpQqfAgka2IcnQEbZzHql\n/LAyo4bWrDyVBaAY2zfj3koYc+4U8iXqCuy/kpOSSuyEcD1+u6HJGixSQgAV43xI\nWBVVM6IQYbWvxb87L6thV8cow9oGJZs/9XPAaeiOkYR+2FJpKAOOPqFEz+WAw/fT\nontVtpcbVBQUleBuCnTDi3cVPX1gOZtNKIrVrbps1qJU+/xjngXLp0a7WPDZTXo8\ngR6Sz6gmqotqc+tBLIHBPTb7K39mhhAcAemvCGj60bi6HnWzEZRsptOCdY2RO655\nyZLa5QepL1CaXi+d7rv1QZSwk18bu6LdZUoaWI8ZXHnfbpiIH79T3/Ekqxt9lcr1\nrvDAP0v2mPJezml4ZXB8zEaPPu+0hxNmeQ3x0rj8BJ/h6EWKo1TotPpFE39R5SJA\nxw000RVV4ZaKcR49dJ3fg+CbsJn9ouIiSswzpDi1IeCfrjZqI6yTp++Sn1UTBRBh\nK10xv55mF5SUPgeaB4D503agVkTGXYnIelq6EC7Pdw==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKgIBAAKCAgEAvflZfI+A6zDuQy2f2CGfgk8p5aDnzyIx58FpZLKnWhq2GaMX\nIlUWBhSNUlwbKJQaeRpAkjubqHJd7HqU6TXkONDd47UfMcbtg8hQMDuw9kT1vpZW\nwdwYbeE9P5jQW9XqBS7dTB7/sla9S3g3UiNvyPDXQL2jU3dxW/4BOI71B6IYi0Ec\nx2HRXZZ67XRiJpjNYFBkzGC5VameFDd1rFwWlLy8zsWOyZP4L/WdIyvFlTToNdPB\n9WaSc5m6UlGdyRRIcNp5ZOw6oP08Aax7FV9ObIVeVWgD6QsRbLDGvZvZztxP/UES\nRTj7OnUlGRxpKpMdctVhnw4bQdb4WFZquCfu22O5c3b6itz0Nur2ZfZ+1T6qB7jh\n1jxRK8fjGxkymaE2dVY84LX4qACzXyZi/Hm6zz+A6HhcPfqqSqH8DDSGKJ8mh921\nr3R3hpdKCe7/XeC14Fp7/upT/vsCcRM2VkcIKMHJRFAhQJsuz5SK/WWRsufqjXWJ\nH4bwcou3IlbjafW2BK1IkCWgXAzbElMMMDkrFJ1iWhiYF3GuUWrZJTuk3BNuDCsg\nv9oHUpXYiGxEmg/tZ9wgGT38GDWb4ObaerVRldbBc2kfMkVlt8jAW4AcoAlePCpw\n3GKQoISkzYNzeJ1JqOq3l16P2mYowO1bQpdHgNa3comrwHUzG3nGAPnOJX0CAwEA\nAQKCAgAT6DKv4kvEmYKGKJxE5jUUTx5ZUTSpWHB5VCOqqSvvPbhSVGxCrT7bEUFD\n3aH87RQfF5inmjZz/tQHyUqca4jISV7PEzY7HZjbA90fSMoA7HYB7DgYJWur6/vi\nk61VykM1GtWWufLaact5QpYtdcgres1bm+OnBe+rj6DqH9TeYWaIBSEyZXYug+vK\nwTTdzYpL6vC9F++j4dvOS4d5scat/F8encYaKnbJ88dtHZOdUId6yH9dK0vycCv9\nn07U+NqkdMhewoBcSV0F4eCd+zg5F/qsIXOzcnT3ekjBKrk2u5BoPe7fsRpmZEso\nZ/k/PF5MH78Uj0Jc8l1gyGFzfb/CL/YZIlMhHFtHR/Jvnk8TRULHiy9R3BVazpiN\n1r33ho90JfE6QscQm3V3y5aJUnla2jhz2QBCUeYXqAqh7CLHRUCjVHZH+pMFunNP\nZ5VQti81qcjHoOlYOqm8hHRnlkPJzvrf6tUzf83Zhx3LwaMTETyQm5ViN8XvuLvA\narB8isbZphM+zagiXIAbvAEDQ4Q38Xgqe4EF+mdP9RgCAbFguiGo/ns10CgRU/2m\naGoTBZnrfzWz93McuPALWBLS9glh2sTFlywpNlTCA3u9kTRFoSpWLo+YpO4TAqIU\np7Urji86TzGSjNJ6iZ8uBBjNlafZ7oEPxnH41CR4wUGHG+qFYQKCAQEA/UPEhXfY\nLur59gfbnNlR2w81PZ7nq/A4xpDeA52RxLOZ94ka/yMmdsyxxBOkbS5e4FI3XL63\nQnuT+efuURYm0rQjm8uPix0sxpvd/HDlar1rpbxo/itvaNAbQC0WzWhKPDpZZ8CL\nbWyHL/W9DdODzh+IcIPVGOTjdtPr/DCHUZbmbB55NHXeoxX0gsnlyEy8CR+2pWAs\nwUNzW3xmgFI/i3AF8QmKeSyJN6+4zBOLMgyW1LI3CqXBNljXeb4/o7T1mQHyoHgf\neiODWy9h+FRQvkyEh0R3rr5LOWU+dgxV55VhyRWkd54wIAcJbx/+d78rs2yhpVkI\nrnUk4J99LELA8QKCAQEAwAaYIfrARJCawYmN9PkCxVMbxPtbu74XCQYtYS/s6Lhc\nPZFNWLKGm4L9MSqcW9lPhyaQGc2tG+IWqJb14VLqcYNP9NcUC0/wc+C0whaTnf2H\n/0Rs8qoOVNI/OjQFk9qA1aQhzx4/wYMV+huVd6PD/gRIbfl2crx+mUvq+3prricx\nAfcUnaryJCM7cc7yF97LeoQ157pBrYSE6l26qTqgbSNtWTpeUECNr2MKogTUzRsO\n53Amoq5J362jfN9IEKYBUteW2peWz5V4BVlReA67dHKuOpBOik40/FRUerlqMaPI\ncsU9uJbFM+6ysVOQX/fhfcOLviRo20oEU4svvsvtTQKCAQEAwzSFd1386YTBLqm1\n0noKRm69WYZuCC86g4nLJTxAyzynEozBmx8rmWuCJKADHwgbpoiYFUsGEx8O2hKy\ntBr/x22C+KT4OYA+L8KZYeZeXVwsaW0D+8vcrgugxWE7bwzQEwXiPT7B+JAqzoN4\nuPxrXeceEaL7D5nz3HXRMOrykadeaL1ALHghJtBYSqp1CmJimCGydFrhhACKHFyQ\n4J0ET2QDoFn7OhRimuI87EawBd6ALuZB2FoTxuDJDBPgFzz2A33l1Vtg35IZv6GQ\nwIe0SEjP6TaTjMMtVcq8Tt6JI9sWqJee0XfCnhzuX5FoYEGf3EaBfOFX+KdfyiYD\no6a78QKCAQEAuDYbP42jYKMwMrXYvscax11g6GkRqVtEEd4hLnihVud1ZYK8pngs\n9YaFieMMuHgR+N8fogFTLcqIXhbjrdKzIU4NKB4kFm4Lh75q6T5rQkbzbgigU3Kb\n6PcAvL8nEp2stkaMQHsOQn+UV6iajHlIDajiKs4DuTHmaP0RYUohfqIOO4ph9pcF\nT3GVFyqlyAxtOw0WAUUGuITGVWaxKKMbP1ebENwTmf0zbqseiisEUfbBJ3jJN4rn\nR3L1wVcZdQ8reeNjzjNZ8625w7fZlcjcP2u8hGXnUnqQDfiASnMQ0qoCI0fgW/IX\nICi+t6ooeEf9w34R0shu5t9soxCFLw3GiQKCAQEAnkTn+t2N9ThYKTR64BAQj6xm\ndF3ZAgmxWY8hkZknTt2acL/Zv4z0qiRok9GA67Uw78t+c2/aGeM5+0BbWhHuAeJX\nih0P7KGREkBAcx4g6qEpxgMS7LNtA9grKGom5jU+PuMdGOy4PRsMJgYNoVzpwjtg\nUAGDghzgJVL5DifvgaiCKeKEI3VtMgEtnnwQ00M3TaQeymgAopXk1oLUMK9W9aDw\n1s0k/VkoX/RI80LBmmjOhjjn8ilnMWdI7ZqX3VfgI43xNwEMnlfNtAmljRr+1Fr4\nGoWw59utYXbAfkwLCWMfg22eMbrKckAkNrpguDzDVuFoTMJYKcG55DoURFS/2g==\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:02 smithi039 ceph-mon[30415]: [21/Sep/2024:00:52:01] ENGINE Serving on https://172.21.15.39:7150 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:02 smithi039 ceph-mon[30415]: [21/Sep/2024:00:52:01] ENGINE Bus STARTED 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:02 smithi039 ceph-mon[30415]: [21/Sep/2024:00:52:01] ENGINE Error in HTTPServer.serve 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: Traceback (most recent call last): 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._connections.run(self.expiration_interval) 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._run(expiration_interval) 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:52:03.273 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: s = self.context.wrap_socket( 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: return self.sslsocket_class._create( 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: self.do_handshake() 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._sslobj.do_handshake() 2024-09-21T00:52:03.274 INFO:journalctl@ceph.mon.a.smithi039.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:52:04.245 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:04 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:04.245 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:04 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:04.928 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T00:52:05.468 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:05 smithi039 ceph-mon[30415]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:06.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T00:52:06.522 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial admin user... 2024-09-21T00:52:07.135 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:06 smithi039 ceph-mon[30415]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:07.136 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:06 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:08.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$RqGlwf9AxgLUJI2.C9mmNOCwMgUISYdSb7q00r58FqlVKNIu4qLj.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726879927, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T00:52:08.236 INFO:teuthology.orchestra.run.smithi039.stdout:Fetching dashboard port number... 2024-09-21T00:52:08.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:08 smithi039 ceph-mon[30415]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:08.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:08 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:09.741 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T00:52:09.761 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T00:52:09.761 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-09-21T00:52:09.776 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T00:52:09.776 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-09-21T00:52:09.776 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-09-21T00:52:09.777 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T00:52:09.780 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph Dashboard is now available at: 2024-09-21T00:52:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: URL: https://smithi039.front.sepia.ceph.com:8443/ 2024-09-21T00:52:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: User: admin 2024-09-21T00:52:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: Password: 51xjms88sm 2024-09-21T00:52:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:09.781 INFO:teuthology.orchestra.run.smithi039.stdout:Saving cluster configuration to /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config directory 2024-09-21T00:52:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:09 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2162916958' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.357 INFO:teuthology.orchestra.run.smithi039.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: ceph telemetry on 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout:For more information see: 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:11.358 INFO:teuthology.orchestra.run.smithi039.stdout:Bootstrap complete. 2024-09-21T00:52:11.416 INFO:tasks.cephadm:Fetching config... 2024-09-21T00:52:11.416 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:52:11.416 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T00:52:11.451 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T00:52:11.452 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:52:11.452 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T00:52:11.528 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T00:52:11.528 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:52:11.528 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T00:52:11.624 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T00:52:11.624 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:52:11.624 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T00:52:11.701 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T00:52:11.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T00:52:11.824 INFO:teuthology.orchestra.run.smithi039.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:11.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T00:52:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:11 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1487365477' entity='client.admin' 2024-09-21T00:52:11.913 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:11.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T00:52:11.981 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOk34gtB4Y6PYBy62iGeSySlcFzb7W9tR8zNfbgFUWM0zewR5Y5a6wkSCZPhbhjFSOkVUgtGbCIhs01bCYfLTURiaTSdkY7GmcMWN60OJYS6y04HyIxd50DVdxVZLiaXCHlJh8BSYx1Ywjx8bcjEim1V/4h3/W1SsamWNRVn9TqPIPpBlfioTvmnOw28igZI3ELxMudL/LQaTvthyTWh/ZHrEtpBr58vA4dyV5YFn8mU1vkLFgHD3bvmP3BHiw7hNpj6AasBRonfj/LqSSIbkXD0EKF/7AXEiF1AWLaFLBjC1CCkNF3zJmXo+4mHL7ufghGwMTJq/UMjoSizI5eqQ+rcdprEYqikBM1qnddwurUOdJeX3RvQhqRmTu/uRQRsJ5o1JWv6q6y+imi2K0mG53hUGXyP+a86WJrrA66yHQ3+qg9MC6SBzW4HqcGK11us5lCDRhPqyABsT3PyFr31tDp4IJZ7HyYBPAvsK18+O7zQTOBnvhpFFatJwOKO9h6Z0= ceph-61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:11.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T00:52:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:12.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:12 smithi039 ceph-mon[30415]: mgrmap e13: x(active, since 12s) 2024-09-21T00:52:14.268 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T00:52:14.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T00:52:14.469 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3137467650' entity='client.admin' 2024-09-21T00:52:14.469 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:14.469 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:14.469 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"} : dispatch 2024-09-21T00:52:14.470 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:14.470 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:14.470 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:14 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:14.877 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:16.834 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2024-09-21T00:52:16.834 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:52:16.834 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2024-09-21T00:52:16.857 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:52:16.857 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:16.918 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2024-09-21T00:52:16.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch host add smithi150 2024-09-21T00:52:17.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:17.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:16 smithi039 ceph-mon[30415]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:52:17.415 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:17 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:17 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:17 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:19.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:18 smithi039 ceph-mon[30415]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:19 smithi039 ceph-mon[30415]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:19 smithi039 ceph-mon[30415]: Deploying cephadm binary to smithi150 2024-09-21T00:52:21.955 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi150' with addr '172.21.15.150' 2024-09-21T00:52:22.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:21 smithi039 ceph-mon[30415]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:22.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:21 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:22.312 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:21 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:22.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T00:52:23.040 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:23.478 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:23 smithi039 ceph-mon[30415]: Added host smithi150 2024-09-21T00:52:23.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:23 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:23.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:23 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:23.868 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2024-09-21T00:52:24.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:24 smithi039 ceph-mon[30415]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:24.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:24 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:24.613 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-09-21T00:52:24.613 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:52:24.613 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-09-21T00:52:24.633 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:52:24.633 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:24.692 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-09-21T00:52:24.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch host add smithi160 2024-09-21T00:52:25.057 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:25.471 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:25 smithi039 ceph-mon[30415]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T00:52:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:26.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:26.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"} : dispatch 2024-09-21T00:52:26.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:26.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:26 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:52:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:52:27.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:27.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:27.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:27 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:28 smithi039 ceph-mon[30415]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:28 smithi039 ceph-mon[30415]: Deploying cephadm binary to smithi160 2024-09-21T00:52:29.697 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-09-21T00:52:30.446 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T00:52:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:30.966 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:30 smithi039 ceph-mon[30415]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:30.966 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:30 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:30.967 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:30 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:30.967 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:30 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:30.967 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:30 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:31.717 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:52:31.718 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-09-21T00:52:31.977 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:31 smithi039 ceph-mon[30415]: Added host smithi160 2024-09-21T00:52:31.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:31 smithi039 ceph-mon[30415]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:31.979 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:31 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:32.455 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T00:52:32.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-21T00:52:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:32.974 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:32 smithi039 ceph-mon[30415]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T00:52:34.624 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2024-09-21T00:52:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1596134194' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T00:52:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"} : dispatch 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:34 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:35.428 INFO:tasks.cephadm:Adding mon.a on smithi039 2024-09-21T00:52:35.429 INFO:tasks.cephadm:Adding mon.b on smithi150 2024-09-21T00:52:35.429 INFO:tasks.cephadm:Adding mon.c on smithi160 2024-09-21T00:52:35.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi039:172.21.15.39=a;smithi150:172.21.15.150=b;smithi160:172.21.15.160=c' 2024-09-21T00:52:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:35 smithi039 ceph-mon[30415]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:35 smithi039 ceph-mon[30415]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:35 smithi039 ceph-mon[30415]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:52:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:35 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1596134194' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T00:52:35.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:35 smithi039 ceph-mon[30415]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:35.678 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:36.324 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2024-09-21T00:52:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:36.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:36.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:36.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:36.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:36.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:36 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:36.910 DEBUG:teuthology.orchestra.run.smithi150:mon.b> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.b.service 2024-09-21T00:52:36.912 DEBUG:teuthology.orchestra.run.smithi160:mon.c> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.c.service 2024-09-21T00:52:36.915 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T00:52:36.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T00:52:37.561 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:37 smithi039 ceph-mon[30415]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi150:172.21.15.150=b;smithi160:172.21.15.160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:37 smithi039 ceph-mon[30415]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi150:172.21.15.150=b;smithi160:172.21.15.160=c;count:3 2024-09-21T00:52:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:37 smithi039 ceph-mon[30415]: Deploying daemon mon.c on smithi160 2024-09-21T00:52:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:52:38.618 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","modified":"2024-09-21T00:50:49.850177Z","created":"2024-09-21T00:50:49.850177Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T00:52:38.618 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-21T00:52:38.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:38 smithi039 ceph-mon[30415]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:38.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:38 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/2697045965' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:40.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:40 smithi039 ceph-mon[30415]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:41.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:40 smithi160 systemd[1]: Started Ceph mon.c for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:52:41.131 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T00:52:41.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T00:52:41.389 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:52:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: mon.a calling monitor election 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: mon.c calling monitor election 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: monmap epoch 2 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: last_changed 2024-09-21T00:52:40.908221+0000 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: min_mon_release 19 (squid) 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: election_strategy: 1 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.c 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: fsmap 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: mgrmap e13: x(active, since 47s) 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: overall HEALTH_OK 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:46.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:46.360 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:46.360 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:45 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:46.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: mon.a calling monitor election 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: mon.c calling monitor election 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:46.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: monmap epoch 2 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: last_changed 2024-09-21T00:52:40.908221+0000 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: min_mon_release 19 (squid) 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: election_strategy: 1 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.c 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: fsmap 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: mgrmap e13: x(active, since 47s) 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: overall HEALTH_OK 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:46.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:46.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:46.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:46.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:45 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:46.499 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:52:46.499 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","modified":"2024-09-21T00:52:40.908221Z","created":"2024-09-21T00:50:49.850177Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T00:52:46.499 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-09-21T00:52:47.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:47 smithi160 ceph-mon[34696]: Deploying daemon mon.b on smithi150 2024-09-21T00:52:47.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:47 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/282006821' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:47.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:47 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:47.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:47 smithi039 ceph-mon[30415]: Deploying daemon mon.b on smithi150 2024-09-21T00:52:47.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:47 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/282006821' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:47.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:47 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:48.229 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T00:52:48.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T00:52:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:48 smithi160 ceph-mon[34696]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:48.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:48 smithi039 ceph-mon[30415]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:48.485 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:52:49.144 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:52:49.144 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","modified":"2024-09-21T00:52:40.908221Z","created":"2024-09-21T00:50:49.850177Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T00:52:49.144 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-09-21T00:52:49.524 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 podman[34927]: 2024-09-21 00:52:49.426267783 +0000 UTC m=+0.015848222 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T00:52:49.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 podman[34927]: 2024-09-21 00:52:49.7307593 +0000 UTC m=+0.320339731 container init 39ea9691389c269a0e1d9058e9f05c03ce1a3930198999b7cb7596d635085032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T00:52:49.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 podman[34927]: 2024-09-21 00:52:49.73398917 +0000 UTC m=+0.323569601 container start 39ea9691389c269a0e1d9058e9f05c03ce1a3930198999b7cb7596d635085032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:52:49.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T00:52:49.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T00:52:49.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pidfile_write: ignore empty --pid-file 2024-09-21T00:52:49.774 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: load: jerasure load: lrc 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: RocksDB version: 7.9.2 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Git sha 0 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: DB SUMMARY 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: DB Session ID: 3PYQOT4FRSBJZ35AAHPJ 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: CURRENT file: CURRENT 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.error_if_exists: 0 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.create_if_missing: 0 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.paranoid_checks: 1 2024-09-21T00:52:49.775 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T00:52:49.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T00:52:49.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T00:52:49.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.env: 0x55ce4fe13d60 2024-09-21T00:52:49.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T00:52:49.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.info_log: 0x55ce55c99360 2024-09-21T00:52:49.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T00:52:49.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.statistics: (nil) 2024-09-21T00:52:49.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.use_fsync: 0 2024-09-21T00:52:49.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_log_file_size: 0 2024-09-21T00:52:49.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T00:52:49.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.allow_fallocate: 1 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.use_direct_reads: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.db_log_dir: 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.wal_dir: 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T00:52:49.778 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.write_buffer_manager: 0x55ce55c9dd60 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T00:52:49.779 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.unordered_write: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.row_cache: None 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.wal_filter: None 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.two_write_queues: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.wal_compression: 0 2024-09-21T00:52:49.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.atomic_flush: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.log_readahead_size: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_background_jobs: 2 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_background_compactions: -1 2024-09-21T00:52:49.781 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_subcompactions: 1 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_open_files: -1 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T00:52:49.782 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_background_flushes: -1 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Compression algorithms supported: 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kZSTD supported: 0 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kXpressCompression supported: 0 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kBZip2Compression supported: 0 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kLZ4Compression supported: 1 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kZlibCompression supported: 1 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: kSnappyCompression supported: 1 2024-09-21T00:52:49.783 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.merge_operator: 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_filter: None 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ce55c98fa0) 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T00:52:49.784 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_type: 0 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_index_type: 0 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_shortening: 1 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: checksum: 4 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: no_block_cache: 0 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache: 0x55ce51c5d610 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_options: 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: capacity : 536870912 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_shard_bits : 4 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: strict_capacity_limit : 0 2024-09-21T00:52:49.785 INFO:journalctl@ceph.mon.b.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_compressed: (nil) 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: persistent_cache: (nil) 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size: 4096 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size_deviation: 10 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_restart_interval: 16 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_block_restart_interval: 1 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: metadata_block_size: 4096 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: partition_filters: 0 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: use_delta_encoding: 1 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: filter_policy: bloomfilter 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: whole_key_filtering: 1 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: verify_compression: 0 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: format_version: 5 2024-09-21T00:52:49.786 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_index_compression: 1 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_align: 0 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout: max_auto_readahead_size: 262144 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout: prepopulate_block_cache: 0 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout: initial_auto_readahead_size: 8192 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression: NoCompression 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.num_levels: 7 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T00:52:49.787 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T00:52:49.788 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T00:52:49.789 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T00:52:49.790 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.inplace_update_support: 0 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.bloom_locality: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.max_successive_merges: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.ttl: 2592000 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.enable_blob_files: false 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.min_blob_size: 0 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d8cf1518-70f4-4857-8698-9d00c9d870e9 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726879969771368, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726879969771810, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726879969, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d8cf1518-70f4-4857-8698-9d00c9d870e9", "db_session_id": "3PYQOT4FRSBJZ35AAHPJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T00:52:49.793 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726879969771908, "job": 1, "event": "recovery_finished"} 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ce55cbca80 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: DB pointer 0x55ce56a62000 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** DB Stats ** 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T00:52:49.794 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T00:52:49.795 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache BinnedLRUCache@0x55ce51c5d610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T00:52:49.796 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T00:52:49.797 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: using public_addr v2:172.21.15.150:0/0 -> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-21T00:52:49.797 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: starting mon.b rank -1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:49.797 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(???) e0 preinit fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T00:52:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T00:52:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout: e1 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout: btime 2024-09-21T00:51:02:310817+0000 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout: legacy client fscid: -1 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout: No filesystems configured 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T00:52:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Added host smithi150 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"} : dispatch 2024-09-21T00:52:50.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:50.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:50.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:50.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:52:50.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:50.067 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Deploying cephadm binary to smithi160 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Added host smithi160 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1596134194' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"} : dispatch 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1596134194' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi150:172.21.15.150=b;smithi160:172.21.15.160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:50.070 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi150:172.21.15.150=b;smithi160:172.21.15.160=c;count:3 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Deploying daemon mon.c on smithi160 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/2697045965' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.a calling monitor election 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.071 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.c calling monitor election 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: monmap epoch 2 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: last_changed 2024-09-21T00:52:40.908221+0000 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: min_mon_release 19 (squid) 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: election_strategy: 1 2024-09-21T00:52:50.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.c 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: fsmap 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mgrmap e13: x(active, since 47s) 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: overall HEALTH_OK 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: Deploying daemon mon.b on smithi150 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/282006821' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:50.073 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 ceph-mon[34945]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T00:52:50.074 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 bash[34927]: 39ea9691389c269a0e1d9058e9f05c03ce1a3930198999b7cb7596d635085032 2024-09-21T00:52:50.074 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:49 smithi150 systemd[1]: Started Ceph mon.b for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:52:50.746 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T00:52:50.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T00:52:50.998 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:52:55.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: mon.a calling monitor election 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: mon.c calling monitor election 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: mon.b calling monitor election 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: monmap epoch 3 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: last_changed 2024-09-21T00:52:49.800006+0000 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: min_mon_release 19 (squid) 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: election_strategy: 1 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.c 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: 2: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: fsmap 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: mgrmap e13: x(active, since 55s) 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: overall HEALTH_OK 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.156 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.156 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:54 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:52:55.285 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":3,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","modified":"2024-09-21T00:52:49.800006Z","created":"2024-09-21T00:50:49.850177Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T00:52:55.286 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 3 2024-09-21T00:52:55.286 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:52:55.286 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.286 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: mon.a calling monitor election 2024-09-21T00:52:55.287 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:55.287 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: mon.c calling monitor election 2024-09-21T00:52:55.287 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.287 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:55.287 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.287 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: mon.b calling monitor election 2024-09-21T00:52:55.288 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.288 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:55.288 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.288 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.288 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T00:52:55.288 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: monmap epoch 3 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: last_changed 2024-09-21T00:52:49.800006+0000 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: min_mon_release 19 (squid) 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: election_strategy: 1 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:52:55.289 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.c 2024-09-21T00:52:55.290 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: 2: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-21T00:52:55.290 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: fsmap 2024-09-21T00:52:55.290 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:55.290 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: mgrmap e13: x(active, since 55s) 2024-09-21T00:52:55.290 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: overall HEALTH_OK 2024-09-21T00:52:55.290 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.291 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.291 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.291 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:54 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:52:55.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: mon.a calling monitor election 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: mon.c calling monitor election 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: mon.b calling monitor election 2024-09-21T00:52:55.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: monmap epoch 3 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:55.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: last_changed 2024-09-21T00:52:49.800006+0000 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: created 2024-09-21T00:50:49.850177+0000 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: min_mon_release 19 (squid) 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: election_strategy: 1 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.c 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: 2: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-21T00:52:55.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: fsmap 2024-09-21T00:52:55.318 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T00:52:55.318 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: mgrmap e13: x(active, since 55s) 2024-09-21T00:52:55.318 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: overall HEALTH_OK 2024-09-21T00:52:55.318 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.318 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.318 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:55.319 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:54 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:56.095 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:55 smithi160 ceph-mon[34696]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:56.096 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:55 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:56.096 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:55 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/4283988360' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:56.096 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:55 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:56.122 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T00:52:56.122 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T00:52:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:55 smithi039 ceph-mon[30415]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:55 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:55 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/4283988360' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:55 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:55 smithi150 ceph-mon[34945]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:55 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:52:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:55 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/4283988360' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T00:52:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:55 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:52:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:52:57.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.257 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.258 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:56 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:56 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:52:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:56 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:57.587 INFO:teuthology.orchestra.run.smithi039.stdout:# minimal ceph.conf for 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:57.587 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-09-21T00:52:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T00:52:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: mon_host = [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-09-21T00:52:58.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: Reconfiguring daemon mon.a on smithi039 2024-09-21T00:52:58.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:52:57 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1356004574' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: Reconfiguring daemon mon.a on smithi039 2024-09-21T00:52:58.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:52:57 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1356004574' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:52:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T00:52:58.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:52:58.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:52:58.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:58.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: Reconfiguring daemon mon.a on smithi039 2024-09-21T00:52:58.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:52:57 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1356004574' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:52:58.871 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T00:52:58.871 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:52:58.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T00:52:58.972 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:52:58.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:59.071 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:52:59.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T00:52:59.105 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:52:59.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:59.174 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:52:59.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T00:52:59.207 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:52:59.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:52:59.277 INFO:tasks.cephadm:enabling module snap_schedule 2024-09-21T00:52:59.277 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-09-21T00:52:59.673 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: Reconfiguring mon.b (monmap changed)... 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:00.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:00.571 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:00.571 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:00 smithi039 ceph-mon[30415]: Reconfiguring daemon mon.b on smithi150 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: Reconfiguring mon.b (monmap changed)... 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:00.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:00 smithi150 ceph-mon[34945]: Reconfiguring daemon mon.b on smithi150 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: Reconfiguring mon.b (monmap changed)... 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:00 smithi160 ceph-mon[34696]: Reconfiguring daemon mon.b on smithi150 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3723455869' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:01.383 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:01 smithi160 ceph-mon[34696]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:01.653 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ignoring --setuser ceph since I am not root 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ignoring --setgroup ceph since I am not root 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3723455869' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:01.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:01 smithi039 ceph-mon[30415]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3723455869' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T00:53:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T00:53:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' 2024-09-21T00:53:01.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:01.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:01.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:01 smithi150 ceph-mon[34945]: from='mgr.14152 172.21.15.39:0/656358464' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:02.231 INFO:tasks.cephadm:Adding mgr.x on smithi039 2024-09-21T00:53:02.231 INFO:tasks.cephadm:Adding mgr.y on smithi150 2024-09-21T00:53:02.231 INFO:tasks.cephadm:Adding mgr.z on smithi160 2024-09-21T00:53:02.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch apply mgr '3;smithi039=x;smithi150=y;smithi160=z' 2024-09-21T00:53:02.473 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:53:02.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:02 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T00:53:02.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:02 smithi039 ceph-mon[30415]: mgrmap e14: x(active, since 62s) 2024-09-21T00:53:02.668 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:02 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T00:53:02.668 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:02 smithi160 ceph-mon[34696]: mgrmap e14: x(active, since 62s) 2024-09-21T00:53:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:02 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T00:53:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:02 smithi150 ceph-mon[34945]: mgrmap e14: x(active, since 62s) 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: Active manager daemon x restarted 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: Activating manager daemon x 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: mgrmap e15: x(active, starting, since 0.00766619s) 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:53:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:53:08.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T00:53:08.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T00:53:08.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T00:53:08.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T00:53:08.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:07 smithi160 ceph-mon[34696]: Manager daemon x is now available 2024-09-21T00:53:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: Active manager daemon x restarted 2024-09-21T00:53:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: Activating manager daemon x 2024-09-21T00:53:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T00:53:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: mgrmap e15: x(active, starting, since 0.00766619s) 2024-09-21T00:53:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:53:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:53:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:53:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T00:53:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T00:53:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T00:53:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T00:53:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:07 smithi039 ceph-mon[30415]: Manager daemon x is now available 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: Active manager daemon x restarted 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: Activating manager daemon x 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: mgrmap e15: x(active, starting, since 0.00766619s) 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:53:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T00:53:08.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T00:53:08.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T00:53:08.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T00:53:08.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:07 smithi150 ceph-mon[34945]: Manager daemon x is now available 2024-09-21T00:53:08.800 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mgr update... 2024-09-21T00:53:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T00:53:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T00:53:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:08 smithi160 ceph-mon[34696]: mgrmap e16: x(active, since 1.01261s) 2024-09-21T00:53:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:09.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:09.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T00:53:09.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T00:53:09.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:08 smithi039 ceph-mon[30415]: mgrmap e16: x(active, since 1.01261s) 2024-09-21T00:53:09.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:09.188 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:09.188 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T00:53:09.188 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T00:53:09.188 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:08 smithi150 ceph-mon[34945]: mgrmap e16: x(active, since 1.01261s) 2024-09-21T00:53:09.188 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:09.397 DEBUG:teuthology.orchestra.run.smithi150:mgr.y> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.y.service 2024-09-21T00:53:09.400 DEBUG:teuthology.orchestra.run.smithi160:mgr.z> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.z.service 2024-09-21T00:53:09.402 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-21T00:53:09.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set client client_mount_timeout 600 2024-09-21T00:53:09.406 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-21T00:53:09.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set client debug_client 20 2024-09-21T00:53:09.450 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-21T00:53:09.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set client debug_ms 1 2024-09-21T00:53:09.498 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-21T00:53:09.498 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set client rados_mon_op_timeout 900 2024-09-21T00:53:09.547 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-21T00:53:09.547 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set client rados_osd_op_timeout 900 2024-09-21T00:53:09.594 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-21T00:53:09.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds 20 2024-09-21T00:53:09.640 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-21T00:53:09.640 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds_balancer 20 2024-09-21T00:53:09.692 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-21T00:53:09.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds debug_ms 2 2024-09-21T00:53:09.742 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-21T00:53:09.742 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-09-21T00:53:09.795 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-09-21T00:53:09.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_merge_size 5 2024-09-21T00:53:09.851 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-09-21T00:53:09.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_bits 3 2024-09-21T00:53:09.901 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-09-21T00:53:09.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_size 100 2024-09-21T00:53:09.962 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-21T00:53:09.962 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_frag True 2024-09-21T00:53:10.014 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-21T00:53:10.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_scatterstat True 2024-09-21T00:53:10.068 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-21T00:53:10.068 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_op_complaint_time 180 2024-09-21T00:53:10.119 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-21T00:53:10.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds mds_verify_scatter True 2024-09-21T00:53:10.177 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-21T00:53:10.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds osd_op_complaint_time 180 2024-09-21T00:53:10.245 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-21T00:53:10.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds rados_mon_op_timeout 900 2024-09-21T00:53:10.306 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-21T00:53:10.306 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mds rados_osd_op_timeout 900 2024-09-21T00:53:10.366 INFO:tasks.cephadm:setting config [mgr] client_mount_timeout = 30 2024-09-21T00:53:10.366 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr client_mount_timeout 30 2024-09-21T00:53:10.426 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-09-21T00:53:10.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr debug_cephsqlite 20 2024-09-21T00:53:10.509 INFO:tasks.cephadm:setting config [mgr] debug_client = 20 2024-09-21T00:53:10.510 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr debug_client 20 2024-09-21T00:53:10.511 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:10.595 INFO:tasks.cephadm:setting config [mgr] debug_mgr = 20 2024-09-21T00:53:10.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr debug_mgr 20 2024-09-21T00:53:10.598 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: Saving service mgr spec with placement smithi039=x;smithi150=y;smithi160=z;count:3 2024-09-21T00:53:10.598 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.598 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.598 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.598 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.599 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.599 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.599 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:10.599 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.599 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.627 INFO:tasks.cephadm:setting config [mgr] debug_ms = 1 2024-09-21T00:53:10.627 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr debug_ms 1 2024-09-21T00:53:10.628 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:10.644 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:10.685 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-09-21T00:53:10.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-09-21T00:53:10.761 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-09-21T00:53:10.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: Saving service mgr spec with placement smithi039=x;smithi150=y;smithi160=z;count:3 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.832 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-09-21T00:53:10.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: Saving service mgr spec with placement smithi039=x;smithi150=y;smithi160=z;count:3 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:10.900 INFO:tasks.cephadm:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-21T00:53:10.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mgr mon_warn_on_pool_no_app False 2024-09-21T00:53:10.901 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:10.987 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-21T00:53:10.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set mon mon_op_complaint_time 120 2024-09-21T00:53:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:11.054 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-21T00:53:11.054 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config set osd osd_op_complaint_time 180 2024-09-21T00:53:11.119 DEBUG:tasks.cephadm:set 30 configs 2024-09-21T00:53:11.119 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi039') 2024-09-21T00:53:11.661 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-mon[30415]: mgrmap e17: x(active, since 2s) 2024-09-21T00:53:11.661 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:11.661 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:11.661 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:11.661 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T00:53:11.468+0000 7f4d9aa00640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:00:53:11] ENGINE Error in HTTPServer.serve 2024-09-21T00:53:11.661 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: Traceback (most recent call last): 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._connections.run(self.expiration_interval) 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._run(expiration_interval) 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: s = self.context.wrap_socket( 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: return self.sslsocket_class._create( 2024-09-21T00:53:11.662 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:53:11.663 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self.do_handshake() 2024-09-21T00:53:11.663 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:53:11.663 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: self._sslobj.do_handshake() 2024-09-21T00:53:11.663 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:53:11.663 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 00:53:11 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T00:53:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:11 smithi150 ceph-mon[34945]: mgrmap e17: x(active, since 2s) 2024-09-21T00:53:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:11 smithi160 ceph-mon[34696]: mgrmap e17: x(active, since 2s) 2024-09-21T00:53:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:12 smithi150 ceph-mon[34945]: [21/Sep/2024:00:53:11] ENGINE Bus STARTING 2024-09-21T00:53:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:12 smithi150 ceph-mon[34945]: [21/Sep/2024:00:53:11] ENGINE Serving on http://172.21.15.39:8765 2024-09-21T00:53:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:12 smithi150 ceph-mon[34945]: [21/Sep/2024:00:53:11] ENGINE Serving on https://172.21.15.39:7150 2024-09-21T00:53:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:12 smithi150 ceph-mon[34945]: [21/Sep/2024:00:53:11] ENGINE Bus STARTED 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:12 smithi150 ceph-mon[34945]: [21/Sep/2024:00:53:11] ENGINE Error in HTTPServer.serve 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: Traceback (most recent call last): 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: self._connections.run(self.expiration_interval) 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: self._run(expiration_interval) 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: s = self.context.wrap_socket( 2024-09-21T00:53:12.815 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout: return self.sslsocket_class._create( 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout: self.do_handshake() 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout: self._sslobj.do_handshake() 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:53:12.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:12 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:12 smithi160 ceph-mon[34696]: [21/Sep/2024:00:53:11] ENGINE Bus STARTING 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:12 smithi160 ceph-mon[34696]: [21/Sep/2024:00:53:11] ENGINE Serving on http://172.21.15.39:8765 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:12 smithi160 ceph-mon[34696]: [21/Sep/2024:00:53:11] ENGINE Serving on https://172.21.15.39:7150 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:12 smithi160 ceph-mon[34696]: [21/Sep/2024:00:53:11] ENGINE Bus STARTED 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:12 smithi160 ceph-mon[34696]: [21/Sep/2024:00:53:11] ENGINE Error in HTTPServer.serve 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout: Traceback (most recent call last): 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:53:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout: self._run(expiration_interval) 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: s = self.context.wrap_socket( 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: return self.sslsocket_class._create( 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: self.do_handshake() 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: self._sslobj.do_handshake() 2024-09-21T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:53:12.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:12 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:12 smithi039 ceph-mon[30415]: [21/Sep/2024:00:53:11] ENGINE Bus STARTING 2024-09-21T00:53:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:12 smithi039 ceph-mon[30415]: [21/Sep/2024:00:53:11] ENGINE Serving on http://172.21.15.39:8765 2024-09-21T00:53:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:12 smithi039 ceph-mon[30415]: [21/Sep/2024:00:53:11] ENGINE Serving on https://172.21.15.39:7150 2024-09-21T00:53:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:12 smithi039 ceph-mon[30415]: [21/Sep/2024:00:53:11] ENGINE Bus STARTED 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:12 smithi039 ceph-mon[30415]: [21/Sep/2024:00:53:11] ENGINE Error in HTTPServer.serve 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: Traceback (most recent call last): 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._connections.run(self.expiration_interval) 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._run(expiration_interval) 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T00:53:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: s = self.context.wrap_socket( 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: return self.sslsocket_class._create( 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: self.do_handshake() 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: self._sslobj.do_handshake() 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T00:53:12.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:12 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:14.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 -- 172.21.15.39:0/3274298510 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9a6403fb70 msgr2=0x7f9a64042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 --2- 172.21.15.39:0/3274298510 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9a6403fb70 0x7f9a64042080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000bbe0 tx=0x7f9a7000b390 comp rx=0 tx=0).stop 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 -- 172.21.15.39:0/3274298510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a7c1092b0 msgr2=0x7f9a7c102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 --2- 172.21.15.39:0/3274298510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a7c1092b0 0x7f9a7c102610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400eb10 tx=0x7f9a7400c2d0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 -- 172.21.15.39:0/3274298510 shutdown_connections 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 --2- 172.21.15.39:0/3274298510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a7c1092b0 0x7f9a7c102610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 --2- 172.21.15.39:0/3274298510 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9a7c106710 0x7f9a7c1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 --2- 172.21.15.39:0/3274298510 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9a6403fb70 0x7f9a64042080 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 --2- 172.21.15.39:0/3274298510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c06c420 0x7f9a7c103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 -- 172.21.15.39:0/3274298510 >> 172.21.15.39:0/3274298510 conn(0x7f9a7c0fde10 msgr2=0x7f9a7c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 -- 172.21.15.39:0/3274298510 shutdown_connections 2024-09-21T00:53:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:13.621+0000 7f9a82200640 1 -- 172.21.15.39:0/3274298510 wait complete. 2024-09-21T00:53:14.292 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2329992058' entity='client.admin' 2024-09-21T00:53:14.292 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3274298510' entity='client.admin' 2024-09-21T00:53:14.292 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:14.292 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:14.292 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:14.293 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:14.293 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 -- 172.21.15.39:0/498684441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b50105180 msgr2=0x7f9b5010bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 --2- 172.21.15.39:0/498684441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b50105180 0x7f9b5010bb20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009ca0 tx=0x7f9b4802a950 comp rx=0 tx=0).stop 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 -- 172.21.15.39:0/498684441 shutdown_connections 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 --2- 172.21.15.39:0/498684441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b50105180 0x7f9b5010bb20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 --2- 172.21.15.39:0/498684441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b501047e0 0x7f9b50104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 --2- 172.21.15.39:0/498684441 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b50103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 -- 172.21.15.39:0/498684441 >> 172.21.15.39:0/498684441 conn(0x7f9b500ff0e0 msgr2=0x7f9b50101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 -- 172.21.15.39:0/498684441 shutdown_connections 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.137+0000 7f9b57e00640 1 -- 172.21.15.39:0/498684441 wait complete. 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.138+0000 7f9b57e00640 1 Processor -- start 2024-09-21T00:53:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.138+0000 7f9b57e00640 1 -- start start 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b57e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b5019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b57e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b501047e0 0x7f9b5019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b57e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b50105180 0x7f9b5019d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b57e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5010f140 con 0x7f9b501047e0 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b57e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5010efc0 con 0x7f9b50105180 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b57e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5010f2c0 con 0x7f9b50103750 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b54a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b501047e0 0x7f9b5019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b5019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b5019bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36364/0 (socket says 172.21.15.39:36364) 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 -- 172.21.15.39:0/3014630021 learned_addr learned my addr 172.21.15.39:0/3014630021 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b50105180 0x7f9b5019d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 -- 172.21.15.39:0/3014630021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b50105180 msgr2=0x7f9b5019d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b50105180 0x7f9b5019d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 -- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b501047e0 msgr2=0x7f9b5019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b501047e0 0x7f9b5019c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.139+0000 7f9b55400640 1 -- 172.21.15.39:0/3014630021 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5019d9f0 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.140+0000 7f9b55400640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b5019bfc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400e9f0 tx=0x7f9b4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.140+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4400cd80 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.140+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f9b440108a0 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.140+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5019dce0 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.140+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b44018750 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.140+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b501a43e0 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.141+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50103bc0 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.142+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f9b440188f0 con 0x7f9b50103750 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.142+0000 7f9b4e000640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9b3803fb70 0x7f9b38042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.142+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f9b44054af0 con 0x7f9b50103750 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.143+0000 7f9b54a00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9b3803fb70 0x7f9b38042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.143+0000 7f9b54a00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9b3803fb70 0x7f9b38042080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0095f0 tx=0x7f9b3c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.147+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9b44020360 con 0x7f9b50103750 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.386+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f9b501a46d0 con 0x7f9b50103750 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.393+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 7 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9b4405a080 con 0x7f9b50103750 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.393+0000 7f9b4e000640 1 -- 172.21.15.39:0/3014630021 <== mon.1 v2:172.21.15.160:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v13)=0 v13) ==== 123+0+0 (secure 0 0 0) 0x7f9b44022d10 con 0x7f9b50103750 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9b3803fb70 msgr2=0x7f9b38042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9b3803fb70 0x7f9b38042080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0095f0 tx=0x7f9b3c005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 msgr2=0x7f9b5019bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b5019bfc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400e9f0 tx=0x7f9b4400eeb0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 shutdown_connections 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9b3803fb70 0x7f9b38042080 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b50105180 0x7f9b5019d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b501047e0 0x7f9b5019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 --2- 172.21.15.39:0/3014630021 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9b50103750 0x7f9b5019bfc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.395+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 >> 172.21.15.39:0/3014630021 conn(0x7f9b500ff0e0 msgr2=0x7f9b500755f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.396+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 shutdown_connections 2024-09-21T00:53:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:14.396+0000 7f9b57e00640 1 -- 172.21.15.39:0/3014630021 wait complete. 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2329992058' entity='client.admin' 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3274298510' entity='client.admin' 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:14.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:14.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2329992058' entity='client.admin' 2024-09-21T00:53:14.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3274298510' entity='client.admin' 2024-09-21T00:53:14.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:14.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:14.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"} : dispatch 2024-09-21T00:53:14.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:14.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:15.534 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:53:15.534 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:53:15.534 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:53:15.534 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.534 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.536 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.536 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:15.536 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T00:53:15.536 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:15.536 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:15.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T00:53:15.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:15.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi039:/etc/ceph/ceph.conf 2024-09-21T00:53:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi150:/etc/ceph/ceph.conf 2024-09-21T00:53:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T00:53:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T00:53:15.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:15.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:15.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-09-21T00:53:15.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:15.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:15.817 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.942+0000 7fbc12200640 1 -- 172.21.15.39:0/534405578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c109e00 msgr2=0x7fbc0c110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.942+0000 7fbc12200640 1 --2- 172.21.15.39:0/534405578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c109e00 0x7fbc0c110630 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009ca0 tx=0x7fbc0402a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 -- 172.21.15.39:0/534405578 shutdown_connections 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 --2- 172.21.15.39:0/534405578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c109e00 0x7fbc0c110630 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 --2- 172.21.15.39:0/534405578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 --2- 172.21.15.39:0/534405578 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbc0c103b70 0x7fbc0c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:15.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 -- 172.21.15.39:0/534405578 >> 172.21.15.39:0/534405578 conn(0x7fbc0c0ff5e0 msgr2=0x7fbc0c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:15.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 -- 172.21.15.39:0/534405578 shutdown_connections 2024-09-21T00:53:15.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.943+0000 7fbc12200640 1 -- 172.21.15.39:0/534405578 wait complete. 2024-09-21T00:53:15.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 Processor -- start 2024-09-21T00:53:15.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 -- start start 2024-09-21T00:53:15.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c103b70 0x7fbc0c19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:15.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:15.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbc0c109e00 0x7fbc0c1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:15.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1139a0 con 0x7fbc0c1094b0 2024-09-21T00:53:15.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c113820 con 0x7fbc0c103b70 2024-09-21T00:53:15.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.944+0000 7fbc12200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c113b20 con 0x7fbc0c109e00 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c19cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38576/0 (socket says 172.21.15.39:38576) 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 -- 172.21.15.39:0/1685511241 learned_addr learned my addr 172.21.15.39:0/1685511241 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0b400640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c103b70 0x7fbc0c19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0be00640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbc0c109e00 0x7fbc0c1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 -- 172.21.15.39:0/1685511241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbc0c109e00 msgr2=0x7fbc0c1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbc0c109e00 0x7fbc0c1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 -- 172.21.15.39:0/1685511241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c103b70 msgr2=0x7fbc0c19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c103b70 0x7fbc0c19c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0aa00640 1 -- 172.21.15.39:0/1685511241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c1a2520 con 0x7fbc0c1094b0 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.945+0000 7fbc0b400640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c103b70 0x7fbc0c19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:15.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.946+0000 7fbc0aa00640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c19cb50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800e9f0 tx=0x7fbbf800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:15.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.946+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf800ccc0 con 0x7fbc0c1094b0 2024-09-21T00:53:15.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.946+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbbf80108c0 con 0x7fbc0c1094b0 2024-09-21T00:53:15.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.946+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c1a2810 con 0x7fbc0c1094b0 2024-09-21T00:53:15.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.946+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8018750 con 0x7fbc0c1094b0 2024-09-21T00:53:15.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.946+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c1a8f70 con 0x7fbc0c1094b0 2024-09-21T00:53:15.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.947+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0c103f80 con 0x7fbc0c1094b0 2024-09-21T00:53:15.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.947+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fbbf80188f0 con 0x7fbc0c1094b0 2024-09-21T00:53:15.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.948+0000 7fbc03e00640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbbf003fb70 0x7fbbf0042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:15.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.948+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fbbf8054a10 con 0x7fbc0c1094b0 2024-09-21T00:53:15.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.948+0000 7fbc0b400640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbbf003fb70 0x7fbbf0042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:15.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.949+0000 7fbc0b400640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbbf003fb70 0x7fbbf0042080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009fc0 tx=0x7fbbfc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:15.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:15.953+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbbf8020360 con 0x7fbc0c1094b0 2024-09-21T00:53:16.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.153+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fbc0c1a9250 con 0x7fbc0c1094b0 2024-09-21T00:53:16.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.162+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7fbbf80071c0 con 0x7fbc0c1094b0 2024-09-21T00:53:16.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.163+0000 7fbc03e00640 1 -- 172.21.15.39:0/1685511241 <== mon.0 v2:172.21.15.39:3300/0 8 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf805a080 con 0x7fbc0c1094b0 2024-09-21T00:53:16.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbbf003fb70 msgr2=0x7fbbf0042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:16.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbbf003fb70 0x7fbbf0042080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009fc0 tx=0x7fbbfc005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:16.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 msgr2=0x7fbc0c19cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:16.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c19cb50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800e9f0 tx=0x7fbbf800eeb0 comp rx=0 tx=0).stop 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 shutdown_connections 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbc0c109e00 0x7fbc0c1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c1094b0 0x7fbc0c19cb50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbbf003fb70 0x7fbbf0042080 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 --2- 172.21.15.39:0/1685511241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc0c103b70 0x7fbc0c19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 >> 172.21.15.39:0/1685511241 conn(0x7fbc0c0ff5e0 msgr2=0x7fbc0c10e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 shutdown_connections 2024-09-21T00:53:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:16.165+0000 7fbc12200640 1 -- 172.21.15.39:0/1685511241 wait complete. 2024-09-21T00:53:16.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:16 smithi039 ceph-mon[30415]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:16.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:16 smithi039 ceph-mon[30415]: Deploying daemon mgr.z on smithi160 2024-09-21T00:53:16.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:16 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1685511241' entity='client.admin' 2024-09-21T00:53:16.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:16 smithi150 ceph-mon[34945]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:16.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:16 smithi150 ceph-mon[34945]: Deploying daemon mgr.z on smithi160 2024-09-21T00:53:16.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:16 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1685511241' entity='client.admin' 2024-09-21T00:53:16.844 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:16 smithi160 ceph-mon[34696]: Updating smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T00:53:16.844 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:16 smithi160 ceph-mon[34696]: Deploying daemon mgr.z on smithi160 2024-09-21T00:53:16.844 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:16 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1685511241' entity='client.admin' 2024-09-21T00:53:16.845 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:16 smithi160 systemd[1]: Starting Ceph mgr.z for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:53:17.107 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:16 smithi160 podman[36640]: 2024-09-21 00:53:16.843932849 +0000 UTC m=+0.111872338 container create 143afbe3a5566adf734851f2bcf669d5407d588c2dc0d70a354e0d0e1958c9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T00:53:17.107 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:16 smithi160 podman[36640]: 2024-09-21 00:53:16.747860743 +0000 UTC m=+0.015800194 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T00:53:17.107 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:17 smithi160 podman[36640]: 2024-09-21 00:53:17.079372634 +0000 UTC m=+0.347312075 container init 143afbe3a5566adf734851f2bcf669d5407d588c2dc0d70a354e0d0e1958c9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-z, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T00:53:17.107 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:17 smithi160 podman[36640]: 2024-09-21 00:53:17.082085638 +0000 UTC m=+0.350025083 container start 143afbe3a5566adf734851f2bcf669d5407d588c2dc0d70a354e0d0e1958c9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-z, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:53:17.385 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:17 smithi160 ceph-mgr[36660]: -- 172.21.15.160:0/1175451262 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b3e04770e0 con 0x55b3e521a000 2024-09-21T00:53:17.385 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:17 smithi160 bash[36640]: 143afbe3a5566adf734851f2bcf669d5407d588c2dc0d70a354e0d0e1958c9e2 2024-09-21T00:53:17.385 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 00:53:17 smithi160 systemd[1]: Started Ceph mgr.z for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:53:17.753 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:17.963 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi039') 2024-09-21T00:53:17.963 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi039') 2024-09-21T00:53:17.963 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi039') 2024-09-21T00:53:17.963 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi039') 2024-09-21T00:53:17.963 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi039') 2024-09-21T00:53:18.068 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.074 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.080 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.080 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.082 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.084 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.089 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.090 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.143 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.148 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.148 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.156 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.170 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.193 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.203 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.207 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.210 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.256 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:18.464 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.464 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.464 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.464 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:18.465 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T00:53:18.465 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:18.465 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:18.465 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:18 smithi039 ceph-mon[30415]: Deploying daemon mgr.y on smithi150 2024-09-21T00:53:18.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:18.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T00:53:18.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:18.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:18.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:18 smithi150 ceph-mon[34945]: Deploying daemon mgr.y on smithi150 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:18.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:18 smithi160 ceph-mon[34696]: Deploying daemon mgr.y on smithi150 2024-09-21T00:53:19.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 -- 172.21.15.39:0/3193550320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c101b60 msgr2=0x7fd40c101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:19.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 --2- 172.21.15.39:0/3193550320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c101b60 0x7fd40c101f70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009900 tx=0x7fd3fc02a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 -- 172.21.15.39:0/3193550320 shutdown_connections 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 --2- 172.21.15.39:0/3193550320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd40c109e10 0x7fd40c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 --2- 172.21.15.39:0/3193550320 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c1094c0 0x7fd40c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 --2- 172.21.15.39:0/3193550320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c101b60 0x7fd40c101f70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 -- 172.21.15.39:0/3193550320 >> 172.21.15.39:0/3193550320 conn(0x7fd40c077800 msgr2=0x7fd40c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 -- 172.21.15.39:0/3193550320 shutdown_connections 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.033+0000 7fd411a00640 1 -- 172.21.15.39:0/3193550320 wait complete. 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.035+0000 7fd411a00640 1 Processor -- start 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.036+0000 7fd411a00640 1 -- start start 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.036+0000 7fd411a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 0x7fd40c19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.036+0000 7fd411a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c1094c0 0x7fd40c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.036+0000 7fd411a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd40c109e10 0x7fd40c19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:19.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.036+0000 7fd411a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c113b00 con 0x7fd40c109e10 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.036+0000 7fd411a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c113980 con 0x7fd40c1094c0 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd411a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c113c80 con 0x7fd40c101b60 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 0x7fd40c19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 0x7fd40c19a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36400/0 (socket says 172.21.15.39:36400) 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 -- 172.21.15.39:0/3289172777 learned_addr learned my addr 172.21.15.39:0/3289172777 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40a000640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c1094c0 0x7fd40c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 -- 172.21.15.39:0/3289172777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c1094c0 msgr2=0x7fd40c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c1094c0 0x7fd40c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 -- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd40c109e10 msgr2=0x7fd40c19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd40c109e10 0x7fd40c19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40aa00640 1 -- 172.21.15.39:0/3289172777 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c1a0410 con 0x7fd40c101b60 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.037+0000 7fd40a000640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c1094c0 0x7fd40c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.038+0000 7fd40aa00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 0x7fd40c19a4d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc02a9e0 tx=0x7fd3fc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:19.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.039+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc03f070 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.039+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3fc031b50 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.039+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c1a06a0 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.039+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc035030 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.039+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40c06cde0 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.041+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c102bd0 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.042+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fd3fc042070 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.043+0000 7fd403400640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd3f003fb70 0x7fd3f0042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.043+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd3fc074cd0 con 0x7fd40c101b60 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.044+0000 7fd40a000640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd3f003fb70 0x7fd3f0042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.045+0000 7fd40a000640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd3f003fb70 0x7fd3f0042080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd40c109380 tx=0x7fd3f8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:19.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.048+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd3fc07a020 con 0x7fd40c101b60 2024-09-21T00:53:19.252 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 systemd[1]: Starting Ceph mgr.y for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:53:19.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.260+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fd40c101f70 con 0x7fd40c101b60 2024-09-21T00:53:19.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.270+0000 7fd403400640 1 -- 172.21.15.39:0/3289172777 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v15)=0 v15) ==== 106+0+0 (secure 0 0 0) 0x7fd3fc03b460 con 0x7fd40c101b60 2024-09-21T00:53:19.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd3f003fb70 msgr2=0x7fd3f0042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:19.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd3f003fb70 0x7fd3f0042080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd40c109380 tx=0x7fd3f8005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:19.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 msgr2=0x7fd40c19a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:19.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 0x7fd40c19a4d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc02a9e0 tx=0x7fd3fc003c10 comp rx=0 tx=0).stop 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 shutdown_connections 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd40c109e10 0x7fd40c19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd3f003fb70 0x7fd3f0042080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c1094c0 0x7fd40c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 --2- 172.21.15.39:0/3289172777 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd40c101b60 0x7fd40c19a4d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.272+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 >> 172.21.15.39:0/3289172777 conn(0x7fd40c077800 msgr2=0x7fd40c100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.273+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 shutdown_connections 2024-09-21T00:53:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.273+0000 7fd411a00640 1 -- 172.21.15.39:0/3289172777 wait complete. 2024-09-21T00:53:19.513 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 podman[36463]: 2024-09-21 00:53:19.252042689 +0000 UTC m=+0.109370217 container create 9814ec731fc18385c65c6540cd622309a41deb552d4b89a4b274c5482bf64ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T00:53:19.513 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 podman[36463]: 2024-09-21 00:53:19.158813714 +0000 UTC m=+0.016141256 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T00:53:19.514 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 podman[36463]: 2024-09-21 00:53:19.513019832 +0000 UTC m=+0.370347359 container init 9814ec731fc18385c65c6540cd622309a41deb552d4b89a4b274c5482bf64ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-y, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T00:53:19.814 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 podman[36463]: 2024-09-21 00:53:19.516186877 +0000 UTC m=+0.373514407 container start 9814ec731fc18385c65c6540cd622309a41deb552d4b89a4b274c5482bf64ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T00:53:19.814 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 ceph-mgr[36482]: -- 172.21.15.150:0/2701745159 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d33a3f0e0 con 0x564d387e2480 2024-09-21T00:53:19.814 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 bash[36463]: 9814ec731fc18385c65c6540cd622309a41deb552d4b89a4b274c5482bf64ce6 2024-09-21T00:53:19.814 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 00:53:19 smithi150 systemd[1]: Started Ceph mgr.y for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:53:20.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 -- 172.21.15.39:0/1140426718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 msgr2=0x7f1fc81061a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 --2- 172.21.15.39:0/1140426718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc81061a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009900 tx=0x7f1fb802a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 -- 172.21.15.39:0/1140426718 shutdown_connections 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 --2- 172.21.15.39:0/1140426718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 0x7f1fc8112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 --2- 172.21.15.39:0/1140426718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 0x7f1fc810bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 --2- 172.21.15.39:0/1140426718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc81061a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.437+0000 7f1fce200640 1 -- 172.21.15.39:0/1140426718 >> 172.21.15.39:0/1140426718 conn(0x7f1fc8101860 msgr2=0x7f1fc8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.438+0000 7f1fce200640 1 -- 172.21.15.39:0/1140426718 shutdown_connections 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.438+0000 7f1fce200640 1 -- 172.21.15.39:0/1140426718 wait complete. 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.439+0000 7f1fce200640 1 Processor -- start 2024-09-21T00:53:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.439+0000 7f1fce200640 1 -- start start 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.439+0000 7f1fce200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.439+0000 7f1fce200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 0x7f1fc819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.439+0000 7f1fce200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 0x7f1fc81a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fce200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8115d30 con 0x7f1fc810b6d0 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fce200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8115bb0 con 0x7f1fc810c020 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fce200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8115eb0 con 0x7f1fc8105d90 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 0x7f1fc819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fc7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fc7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 0x7f1fc81a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fc7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36412/0 (socket says 172.21.15.39:36412) 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fc7400640 1 -- 172.21.15.39:0/2492331659 learned_addr learned my addr 172.21.15.39:0/2492331659 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:20.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fbfe00640 1 -- 172.21.15.39:0/2492331659 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 msgr2=0x7f1fc819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fbfe00640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fbfe00640 1 -- 172.21.15.39:0/2492331659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 msgr2=0x7f1fc81a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fbfe00640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 0x7f1fc81a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fbfe00640 1 -- 172.21.15.39:0/2492331659 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc81a2730 con 0x7f1fc810b6d0 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fc7400640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.440+0000 7f1fc7e00640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 0x7f1fc81a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.441+0000 7f1fbfe00640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 0x7f1fc819cd60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1fc806ca80 tx=0x7f1fb000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.441+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb000cc40 con 0x7f1fc810b6d0 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.441+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb000cde0 con 0x7f1fc810b6d0 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.441+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0018780 con 0x7f1fc810b6d0 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.441+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc81a2a20 con 0x7f1fc810b6d0 2024-09-21T00:53:20.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.441+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc81a9180 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.443+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc81061a0 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.443+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f1fb0010430 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.443+0000 7f1fc4c00640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1fac03fb70 0x7f1fac042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.443+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1fb0054a10 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.444+0000 7f1fc7400640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1fac03fb70 0x7f1fac042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.445+0000 7f1fc7400640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1fac03fb70 0x7f1fac042080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1fb80096c0 tx=0x7f1fb8005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.449+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1fb0010720 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.650+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f1fc810ddf0 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.659+0000 7f1fc4c00640 1 -- 172.21.15.39:0/2492331659 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v16)=0 v16) ==== 117+0+0 (secure 0 0 0) 0x7f1fb0010a90 con 0x7f1fc810b6d0 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.661+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1fac03fb70 msgr2=0x7f1fac042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.661+0000 7f1fce200640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1fac03fb70 0x7f1fac042080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1fb80096c0 tx=0x7f1fb8005a20 comp rx=0 tx=0).stop 2024-09-21T00:53:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.661+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 msgr2=0x7f1fc819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.661+0000 7f1fce200640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 0x7f1fc819cd60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1fc806ca80 tx=0x7f1fb000ee50 comp rx=0 tx=0).stop 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 shutdown_connections 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1fac03fb70 0x7f1fac042080 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fc810c020 0x7f1fc81a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fc810b6d0 0x7f1fc819cd60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 --2- 172.21.15.39:0/2492331659 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1fc8105d90 0x7f1fc819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 >> 172.21.15.39:0/2492331659 conn(0x7f1fc8101860 msgr2=0x7f1fc810ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 shutdown_connections 2024-09-21T00:53:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:19.662+0000 7f1fce200640 1 -- 172.21.15.39:0/2492331659 wait complete. 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2492331659' entity='client.admin' 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2492331659' entity='client.admin' 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.549 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2492331659' entity='client.admin' 2024-09-21T00:53:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:20.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 -- 172.21.15.39:0/3572407270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 msgr2=0x7f65ec10f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 --2- 172.21.15.39:0/3572407270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 0x7f65ec10f9d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009900 tx=0x7f65d802a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 -- 172.21.15.39:0/3572407270 shutdown_connections 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 --2- 172.21.15.39:0/3572407270 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 0x7f65ec112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 --2- 172.21.15.39:0/3572407270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 0x7f65ec10f9d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 --2- 172.21.15.39:0/3572407270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 -- 172.21.15.39:0/3572407270 >> 172.21.15.39:0/3572407270 conn(0x7f65ec0ff440 msgr2=0x7f65ec101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 -- 172.21.15.39:0/3572407270 shutdown_connections 2024-09-21T00:53:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.131+0000 7f65f1e00640 1 -- 172.21.15.39:0/3572407270 wait complete. 2024-09-21T00:53:21.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.132+0000 7f65f1e00640 1 Processor -- start 2024-09-21T00:53:21.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 -- start start 2024-09-21T00:53:21.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 0x7f65ec19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 0x7f65ec1a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec115ac0 con 0x7f65ec076100 2024-09-21T00:53:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec115940 con 0x7f65ec0757b0 2024-09-21T00:53:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f1e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec115c40 con 0x7f65ec10ff10 2024-09-21T00:53:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.133+0000 7f65f0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec19c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46618/0 (socket says 172.21.15.39:46618) 2024-09-21T00:53:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 -- 172.21.15.39:0/2561873256 learned_addr learned my addr 172.21.15.39:0/2561873256 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65ebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 0x7f65ec19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f1400640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 0x7f65ec1a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 -- 172.21.15.39:0/2561873256 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 msgr2=0x7f65ec1a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 0x7f65ec1a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 -- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 msgr2=0x7f65ec19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 0x7f65ec19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 -- 172.21.15.39:0/2561873256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ec1a2460 con 0x7f65ec0757b0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f1400640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 0x7f65ec1a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.134+0000 7f65f0a00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec19c4f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f65e000ea60 tx=0x7f65e000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.135+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e000cde0 con 0x7f65ec0757b0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.135+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65e00035f0 con 0x7f65ec0757b0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.135+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e0010690 con 0x7f65ec0757b0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.135+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ec1a2750 con 0x7f65ec0757b0 2024-09-21T00:53:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.135+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ec1a8eb0 con 0x7f65ec0757b0 2024-09-21T00:53:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.136+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ec075bc0 con 0x7f65ec0757b0 2024-09-21T00:53:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.136+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f65e0010830 con 0x7f65ec0757b0 2024-09-21T00:53:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.137+0000 7f65e9600640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65d403fb70 0x7f65d4042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.137+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f65e0054960 con 0x7f65ec0757b0 2024-09-21T00:53:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.137+0000 7f65ebe00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65d403fb70 0x7f65d4042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:21.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.138+0000 7f65ebe00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65d403fb70 0x7f65d4042080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f65d8000c00 tx=0x7f65d8031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:21.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.142+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f65e0003810 con 0x7f65ec0757b0 2024-09-21T00:53:21.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.375+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f65ec1a91e0 con 0x7f65ec0757b0 2024-09-21T00:53:21.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.385+0000 7f65e9600640 1 -- 172.21.15.39:0/2561873256 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v19)=0 v19) ==== 116+0+0 (secure 0 0 0) 0x7f65e0003ac0 con 0x7f65ec0757b0 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65d403fb70 msgr2=0x7f65d4042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65d403fb70 0x7f65d4042080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f65d8000c00 tx=0x7f65d8031400 comp rx=0 tx=0).stop 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 msgr2=0x7f65ec19c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec19c4f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f65e000ea60 tx=0x7f65e000ef20 comp rx=0 tx=0).stop 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 shutdown_connections 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65ec10ff10 0x7f65ec1a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ec076100 0x7f65ec19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65d403fb70 0x7f65d4042080 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 --2- 172.21.15.39:0/2561873256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec0757b0 0x7f65ec19c4f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:21.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.388+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 >> 172.21.15.39:0/2561873256 conn(0x7f65ec0ff440 msgr2=0x7f65ec109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:21.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.389+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 shutdown_connections 2024-09-21T00:53:21.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.389+0000 7f65f1e00640 1 -- 172.21.15.39:0/2561873256 wait complete. 2024-09-21T00:53:21.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:21 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:21.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:21.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:21.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:21 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T00:53:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T00:53:21.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:21.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:21.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:21 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:22.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T00:53:22.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: Reconfiguring daemon mgr.x on smithi039 2024-09-21T00:53:22.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:22.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: Standby manager daemon z started 2024-09-21T00:53:22.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T00:53:22.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T00:53:22.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T00:53:22.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:22 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T00:53:22.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 -- 172.21.15.39:0/615157912 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 msgr2=0x7f51bc103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:22.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 --2- 172.21.15.39:0/615157912 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc103f70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f51ac009860 tx=0x7f51ac02a340 comp rx=0 tx=0).stop 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 -- 172.21.15.39:0/615157912 shutdown_connections 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 --2- 172.21.15.39:0/615157912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51bc109df0 0x7f51bc110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 --2- 172.21.15.39:0/615157912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc1094a0 0x7f51bc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 --2- 172.21.15.39:0/615157912 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc103f70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 -- 172.21.15.39:0/615157912 >> 172.21.15.39:0/615157912 conn(0x7f51bc07b850 msgr2=0x7f51bc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.256+0000 7f51c0e00640 1 -- 172.21.15.39:0/615157912 shutdown_connections 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.257+0000 7f51c0e00640 1 -- 172.21.15.39:0/615157912 wait complete. 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 Processor -- start 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 -- start start 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51bc1094a0 0x7f51bc19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc109df0 0x7f51bc1a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:22.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc113ab0 con 0x7f51bc1094a0 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc113930 con 0x7f51bc109df0 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.258+0000 7f51c0e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc113c30 con 0x7f51bc103b60 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51bc1094a0 0x7f51bc19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51ba800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc109df0 0x7f51bc1a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36424/0 (socket says 172.21.15.39:36424) 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 -- 172.21.15.39:0/1544275657 learned_addr learned my addr 172.21.15.39:0/1544275657 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 -- 172.21.15.39:0/1544275657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc109df0 msgr2=0x7f51bc1a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc109df0 0x7f51bc1a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 -- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51bc1094a0 msgr2=0x7f51bc19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51bc1094a0 0x7f51bc19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 -- 172.21.15.39:0/1544275657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51bc1a2730 con 0x7f51bc103b60 2024-09-21T00:53:22.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51ba800640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc109df0 0x7f51bc1a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.259+0000 7f51b9e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc19c7c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0303f0 tx=0x7f51ac032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.260+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac040020 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.260+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51ac030dd0 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.260+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51bc1a29c0 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.260+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac037040 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.260+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51bc1a9180 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.261+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51bc103f70 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.261+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f51ac043020 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.262+0000 7f51b2a00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51a003fb70 0x7f51a0042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.262+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f51ac074290 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.262+0000 7f51b9400640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51a003fb70 0x7f51a0042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.262+0000 7f51b9400640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51a003fb70 0x7f51a0042080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f51bc108fe0 tx=0x7f51a8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.266+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f51ac04a020 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.477+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f51bc1a9460 con 0x7f51bc103b60 2024-09-21T00:53:22.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.485+0000 7f51b2a00640 1 -- 172.21.15.39:0/1544275657 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v17)=0 v17) ==== 133+0+0 (secure 0 0 0) 0x7f51ac0328c0 con 0x7f51bc103b60 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51a003fb70 msgr2=0x7f51a0042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51a003fb70 0x7f51a0042080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f51bc108fe0 tx=0x7f51a8005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 msgr2=0x7f51bc19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc19c7c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0303f0 tx=0x7f51ac032da0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 shutdown_connections 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51a003fb70 0x7f51a0042080 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.487+0000 7f51c0e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51bc109df0 0x7f51bc1a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.488+0000 7f51c0e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51bc1094a0 0x7f51bc19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.488+0000 7f51c0e00640 1 --2- 172.21.15.39:0/1544275657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51bc103b60 0x7f51bc19c7c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.488+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 >> 172.21.15.39:0/1544275657 conn(0x7f51bc07b850 msgr2=0x7f51bc1073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.488+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 shutdown_connections 2024-09-21T00:53:22.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:20.488+0000 7f51c0e00640 1 -- 172.21.15.39:0/1544275657 wait complete. 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: Reconfiguring daemon mgr.x on smithi039 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: Standby manager daemon z started 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T00:53:22.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:22 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: Reconfiguring daemon mgr.x on smithi039 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: Standby manager daemon z started 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T00:53:22.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:22 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.160:0/459536991' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T00:53:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 -- 172.21.15.39:0/3101307716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 msgr2=0x7f4eec109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/3101307716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec109700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009900 tx=0x7f4ed802a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 -- 172.21.15.39:0/3101307716 shutdown_connections 2024-09-21T00:53:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/3101307716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec109c40 0x7f4eec110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/3101307716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec109700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/3101307716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4eec1039b0 0x7f4eec103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 -- 172.21.15.39:0/3101307716 >> 172.21.15.39:0/3101307716 conn(0x7f4eec0ff440 msgr2=0x7f4eec101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 -- 172.21.15.39:0/3101307716 shutdown_connections 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.761+0000 7f4ef0a00640 1 -- 172.21.15.39:0/3101307716 wait complete. 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.762+0000 7f4ef0a00640 1 Processor -- start 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.762+0000 7f4ef0a00640 1 -- start start 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.762+0000 7f4ef0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 0x7f4eec19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.762+0000 7f4ef0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4ef0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4eec109c40 0x7f4eec19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4ef0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec113950 con 0x7f4eec1092f0 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4ef0a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1137d0 con 0x7f4eec109c40 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4ef0a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec113ad0 con 0x7f4eec1039b0 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 0x7f4eec19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 0x7f4eec19a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36468/0 (socket says 172.21.15.39:36468) 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 -- 172.21.15.39:0/1899314969 learned_addr learned my addr 172.21.15.39:0/1899314969 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eebe00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4eec109c40 0x7f4eec19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 -- 172.21.15.39:0/1899314969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4eec109c40 msgr2=0x7f4eec19fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4eec109c40 0x7f4eec19fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 -- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 msgr2=0x7f4eec19a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec19a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeb400640 1 -- 172.21.15.39:0/1899314969 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec1a0270 con 0x7f4eec1039b0 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.763+0000 7f4eeaa00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.764+0000 7f4eeb400640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 0x7f4eec19a300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00ea60 tx=0x7f4edc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.764+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc00cde0 con 0x7f4eec1039b0 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.764+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eec1a0560 con 0x7f4eec1039b0 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.764+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4edc0035f0 con 0x7f4eec1039b0 2024-09-21T00:53:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.764+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc010690 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.764+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eec1a6cc0 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.765+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eec103dc0 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.765+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f4edc004020 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.766+0000 7f4ee3e00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4ed003fb70 0x7f4ed0042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.766+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4edc054920 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.766+0000 7f4eeaa00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4ed003fb70 0x7f4ed0042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.767+0000 7f4eeaa00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4ed003fb70 0x7f4ed0042080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4ed802a9e0 tx=0x7f4ed8005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.771+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4edc003ac0 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.976+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f4eec02e640 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.982+0000 7f4ee3e00640 1 -- 172.21.15.39:0/1899314969 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v20)=0 v20) ==== 127+0+0 (secure 0 0 0) 0x7f4edc021cd0 con 0x7f4eec1039b0 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.984+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4ed003fb70 msgr2=0x7f4ed0042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.984+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4ed003fb70 0x7f4ed0042080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4ed802a9e0 tx=0x7f4ed8005a20 comp rx=0 tx=0).stop 2024-09-21T00:53:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 msgr2=0x7f4eec19a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 0x7f4eec19a300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00ea60 tx=0x7f4edc00ef20 comp rx=0 tx=0).stop 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 shutdown_connections 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4ed003fb70 0x7f4ed0042080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4eec109c40 0x7f4eec19fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4eec1092f0 0x7f4eec19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 --2- 172.21.15.39:0/1899314969 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4eec1039b0 0x7f4eec19a300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 >> 172.21.15.39:0/1899314969 conn(0x7f4eec0ff440 msgr2=0x7f4eec101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 shutdown_connections 2024-09-21T00:53:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.985+0000 7f4ef0a00640 1 -- 172.21.15.39:0/1899314969 wait complete. 2024-09-21T00:53:23.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:23 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:23.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:23 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:23.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:23 smithi039 ceph-mon[30415]: mgrmap e18: x(active, since 15s), standbys: z 2024-09-21T00:53:23.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T00:53:23.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:23 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:23 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:23 smithi160 ceph-mon[34696]: mgrmap e18: x(active, since 15s), standbys: z 2024-09-21T00:53:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T00:53:23.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:23 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:23 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:23 smithi150 ceph-mon[34945]: mgrmap e18: x(active, since 15s), standbys: z 2024-09-21T00:53:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T00:53:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:24.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:24 smithi039 ceph-mon[30415]: Standby manager daemon y started 2024-09-21T00:53:24.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:24 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T00:53:24.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:24 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T00:53:24.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:24 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T00:53:24.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:24 smithi039 ceph-mon[30415]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T00:53:24.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 -- 172.21.15.39:0/3386166402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e81094a0 msgr2=0x7f50e81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:24.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 --2- 172.21.15.39:0/3386166402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e81094a0 0x7f50e81098b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009900 tx=0x7f50d402a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:24.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 -- 172.21.15.39:0/3386166402 shutdown_connections 2024-09-21T00:53:24.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 --2- 172.21.15.39:0/3386166402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e8109df0 0x7f50e8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 --2- 172.21.15.39:0/3386166402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e81094a0 0x7f50e81098b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 --2- 172.21.15.39:0/3386166402 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f50e8103b60 0x7f50e8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 -- 172.21.15.39:0/3386166402 >> 172.21.15.39:0/3386166402 conn(0x7f50e807b850 msgr2=0x7f50e807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 -- 172.21.15.39:0/3386166402 shutdown_connections 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.904+0000 7f50f0600640 1 -- 172.21.15.39:0/3386166402 wait complete. 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 Processor -- start 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 -- start start 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 0x7f50e819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e81094a0 0x7f50e819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f50e8109df0 0x7f50e819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f50e8113b00 con 0x7f50e8103b60 2024-09-21T00:53:24.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f50e8113980 con 0x7f50e81094a0 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.905+0000 7f50f0600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f50e8113c80 con 0x7f50e8109df0 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 0x7f50e819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 0x7f50e819a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60646/0 (socket says 172.21.15.39:60646) 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 -- 172.21.15.39:0/1407835460 learned_addr learned my addr 172.21.15.39:0/1407835460 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50ee600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f50e8109df0 0x7f50e819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50ed200640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e81094a0 0x7f50e819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 -- 172.21.15.39:0/1407835460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f50e8109df0 msgr2=0x7f50e819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f50e8109df0 0x7f50e819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 -- 172.21.15.39:0/1407835460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e81094a0 msgr2=0x7f50e819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:24.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e81094a0 0x7f50e819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.906+0000 7f50edc00640 1 -- 172.21.15.39:0/1407835460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e81a03e0 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.907+0000 7f50edc00640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 0x7f50e819a470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00d950 tx=0x7f50dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.907+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc017070 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.907+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e81a06d0 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.907+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50dc009c30 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.907+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00f620 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.907+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e81a6e30 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.908+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e8103f70 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.912+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f50dc005030 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.914+0000 7f50e6000640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50d00b64d0 0x7f50d00b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.914+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f50dc0e5280 con 0x7f50e8103b60 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.914+0000 7f50ed200640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50d00b64d0 0x7f50d00b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:24.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.915+0000 7f50ed200640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50d00b64d0 0x7f50d00b89e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f50d402a9e0 tx=0x7f50d4005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:24.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.916+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f50dc009de0 con 0x7f50e8103b60 2024-09-21T00:53:25.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.379+0000 7f37c5400640 1 -- 172.21.15.39:0/2080013795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c00760a0 msgr2=0x7f37c010fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.379+0000 7f37c5400640 1 --2- 172.21.15.39:0/2080013795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c00760a0 0x7f37c010fa30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009860 tx=0x7f37ac02a340 comp rx=0 tx=0).stop 2024-09-21T00:53:25.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.380+0000 7f37c5400640 1 -- 172.21.15.39:0/2080013795 shutdown_connections 2024-09-21T00:53:25.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.380+0000 7f37c5400640 1 --2- 172.21.15.39:0/2080013795 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c010ff70 0x7f37c0112390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.380+0000 7f37c5400640 1 --2- 172.21.15.39:0/2080013795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c00760a0 0x7f37c010fa30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.380+0000 7f37c5400640 1 --2- 172.21.15.39:0/2080013795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c0075750 0x7f37c0075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.380+0000 7f37c5400640 1 -- 172.21.15.39:0/2080013795 >> 172.21.15.39:0/2080013795 conn(0x7f37c00ff400 msgr2=0x7f37c0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.381+0000 7f37c5400640 1 -- 172.21.15.39:0/2080013795 shutdown_connections 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.381+0000 7f37c5400640 1 -- 172.21.15.39:0/2080013795 wait complete. 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.382+0000 7f37c5400640 1 Processor -- start 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.382+0000 7f37c5400640 1 -- start start 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.382+0000 7f37c5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0075750 0x7f37c019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.382+0000 7f37c5400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 0x7f37c019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37c5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c010ff70 0x7f37c01a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37c5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f37c0115aa0 con 0x7f37c0075750 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37c5400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f37c0115920 con 0x7f37c010ff70 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37c5400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f37c0115c20 con 0x7f37c00760a0 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0075750 0x7f37c019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 0x7f37c019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 0x7f37c019cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36480/0 (socket says 172.21.15.39:36480) 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 -- 172.21.15.39:0/397360873 learned_addr learned my addr 172.21.15.39:0/397360873 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0075750 0x7f37c019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38692/0 (socket says 172.21.15.39:38692) 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37c4a00640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c010ff70 0x7f37c01a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 -- 172.21.15.39:0/397360873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c010ff70 msgr2=0x7f37c01a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c010ff70 0x7f37c01a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 -- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0075750 msgr2=0x7f37c019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0075750 0x7f37c019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.383+0000 7f37bf400640 1 -- 172.21.15.39:0/397360873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c01a23c0 con 0x7f37c00760a0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.384+0000 7f37bf400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 0x7f37c019cab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37ac005940 tx=0x7f37ac032eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.384+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac040020 con 0x7f37c00760a0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.384+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37ac030450 con 0x7f37c00760a0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.384+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c01a2650 con 0x7f37c00760a0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.384+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac037040 con 0x7f37c00760a0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.384+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c01a2b30 con 0x7f37c00760a0 2024-09-21T00:53:25.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.386+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f37ac043020 con 0x7f37c00760a0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.386+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c0075b60 con 0x7f37c00760a0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.386+0000 7f37bcc00640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f37a403fba0 0x7f37a40420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.386+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f37ac030600 con 0x7f37c00760a0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.387+0000 7f37bfe00640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f37a403fba0 0x7f37a40420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.387+0000 7f37bfe00640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f37a403fba0 0x7f37a40420b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37b00045f0 tx=0x7f37b000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.391+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f37ac04a020 con 0x7f37c00760a0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.595+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f37c01a8fc0 con 0x7f37c00760a0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.601+0000 7f37bcc00640 1 -- 172.21.15.39:0/397360873 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v21)=0 v21) ==== 120+0+0 (secure 0 0 0) 0x7f37ac000ed0 con 0x7f37c00760a0 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f37a403fba0 msgr2=0x7f37a40420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f37a403fba0 0x7f37a40420b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37b00045f0 tx=0x7f37b000b390 comp rx=0 tx=0).stop 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 msgr2=0x7f37c019cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 0x7f37c019cab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37ac005940 tx=0x7f37ac032eb0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 shutdown_connections 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f37a403fba0 0x7f37a40420b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c010ff70 0x7f37c01a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f37c00760a0 0x7f37c019cab0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 --2- 172.21.15.39:0/397360873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0075750 0x7f37c019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 >> 172.21.15.39:0/397360873 conn(0x7f37c00ff400 msgr2=0x7f37c0108a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 shutdown_connections 2024-09-21T00:53:25.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.604+0000 7f37c5400640 1 -- 172.21.15.39:0/397360873 wait complete. 2024-09-21T00:53:25.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:24 smithi150 ceph-mon[34945]: Standby manager daemon y started 2024-09-21T00:53:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:24 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T00:53:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:24 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T00:53:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:24 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T00:53:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:24 smithi150 ceph-mon[34945]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T00:53:25.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:24 smithi160 ceph-mon[34696]: Standby manager daemon y started 2024-09-21T00:53:25.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:24 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T00:53:25.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:24 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T00:53:25.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:24 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T00:53:25.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:24 smithi160 ceph-mon[34696]: from='mgr.? 172.21.15.150:0/45068047' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T00:53:25.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.122+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f50e819b5a0 con 0x7f50e8103b60 2024-09-21T00:53:25.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.127+0000 7f50e6000640 1 -- 172.21.15.39:0/1407835460 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v23)=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f50dc0b1af0 con 0x7f50e8103b60 2024-09-21T00:53:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50d00b64d0 msgr2=0x7f50d00b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50d00b64d0 0x7f50d00b89e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f50d402a9e0 tx=0x7f50d4005a20 comp rx=0 tx=0).stop 2024-09-21T00:53:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 msgr2=0x7f50e819a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 0x7f50e819a470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00d950 tx=0x7f50dc00de10 comp rx=0 tx=0).stop 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 shutdown_connections 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f50e8109df0 0x7f50e819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50d00b64d0 0x7f50d00b89e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e81094a0 0x7f50e819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 --2- 172.21.15.39:0/1407835460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e8103b60 0x7f50e819a470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 >> 172.21.15.39:0/1407835460 conn(0x7f50e807b850 msgr2=0x7f50e81054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 shutdown_connections 2024-09-21T00:53:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.130+0000 7f50f0600640 1 -- 172.21.15.39:0/1407835460 wait complete. 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.989+0000 7fb541800640 1 -- 172.21.15.39:0/3142764365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c1100e0 msgr2=0x7fb53c112520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.989+0000 7fb541800640 1 --2- 172.21.15.39:0/3142764365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c1100e0 0x7fb53c112520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb534009ca0 tx=0x7fb53402a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 -- 172.21.15.39:0/3142764365 shutdown_connections 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 --2- 172.21.15.39:0/3142764365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c1100e0 0x7fb53c112520 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 --2- 172.21.15.39:0/3142764365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c10fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 --2- 172.21.15.39:0/3142764365 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb53c0757b0 0x7fb53c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 -- 172.21.15.39:0/3142764365 >> 172.21.15.39:0/3142764365 conn(0x7fb53c0ff590 msgr2=0x7fb53c1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 -- 172.21.15.39:0/3142764365 shutdown_connections 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.990+0000 7fb541800640 1 -- 172.21.15.39:0/3142764365 wait complete. 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 Processor -- start 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 -- start start 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c0757b0 0x7fb53c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb53c1100e0 0x7fb53c1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c115c90 con 0x7fb53c076100 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c115b10 con 0x7fb53c0757b0 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.991+0000 7fb541800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c115e10 con 0x7fb53c1100e0 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60664/0 (socket says 172.21.15.39:60664) 2024-09-21T00:53:25.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 -- 172.21.15.39:0/616833471 learned_addr learned my addr 172.21.15.39:0/616833471 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53aa00640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c0757b0 0x7fb53c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53b400640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb53c1100e0 0x7fb53c1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 -- 172.21.15.39:0/616833471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb53c1100e0 msgr2=0x7fb53c1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb53c1100e0 0x7fb53c1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 -- 172.21.15.39:0/616833471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c0757b0 msgr2=0x7fb53c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c0757b0 0x7fb53c19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 -- 172.21.15.39:0/616833471 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c1a2590 con 0x7fb53c076100 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53aa00640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c0757b0 0x7fb53c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.992+0000 7fb53a000640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c19cbc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e9c0 tx=0x7fb52800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.993+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52800cd80 con 0x7fb53c076100 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.993+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5280035f0 con 0x7fb53c076100 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.993+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c1a2880 con 0x7fb53c076100 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.993+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528010690 con 0x7fb53c076100 2024-09-21T00:53:25.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.993+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c1a8fe0 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.994+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53c075bc0 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.996+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fb5280037a0 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.998+0000 7fb533400640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb5200b64d0 0x7fb5200b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.998+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb5280e5c20 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.998+0000 7fb53aa00640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb5200b64d0 0x7fb5200b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:24.999+0000 7fb53aa00640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb5200b64d0 0x7fb5200b89e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009fc0 tx=0x7fb52c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.000+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb5280ae010 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.212+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fb53c1a92c0 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.220+0000 7fb533400640 1 -- 172.21.15.39:0/616833471 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v24)=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7fb5280b23f0 con 0x7fb53c076100 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.222+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb5200b64d0 msgr2=0x7fb5200b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb5200b64d0 0x7fb5200b89e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009fc0 tx=0x7fb52c005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 msgr2=0x7fb53c19cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c19cbc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e9c0 tx=0x7fb52800ee80 comp rx=0 tx=0).stop 2024-09-21T00:53:25.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 shutdown_connections 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb53c1100e0 0x7fb53c1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb53c076100 0x7fb53c19cbc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb5200b64d0 0x7fb5200b89e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 --2- 172.21.15.39:0/616833471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53c0757b0 0x7fb53c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 >> 172.21.15.39:0/616833471 conn(0x7fb53c0ff590 msgr2=0x7fb53c101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 shutdown_connections 2024-09-21T00:53:25.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:25.223+0000 7fb541800640 1 -- 172.21.15.39:0/616833471 wait complete. 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 -- 172.21.15.39:0/2812814821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 msgr2=0x7fa80810fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 --2- 172.21.15.39:0/2812814821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80810fba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009900 tx=0x7fa7f002a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 -- 172.21.15.39:0/2812814821 shutdown_connections 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 --2- 172.21.15.39:0/2812814821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8081100e0 0x7fa808112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 --2- 172.21.15.39:0/2812814821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80810fba0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 --2- 172.21.15.39:0/2812814821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8080757b0 0x7fa808075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.721+0000 7fa80f200640 1 -- 172.21.15.39:0/2812814821 >> 172.21.15.39:0/2812814821 conn(0x7fa8080ff610 msgr2=0x7fa808101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.722+0000 7fa80f200640 1 -- 172.21.15.39:0/2812814821 shutdown_connections 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.722+0000 7fa80f200640 1 -- 172.21.15.39:0/2812814821 wait complete. 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.722+0000 7fa80f200640 1 Processor -- start 2024-09-21T00:53:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 -- start start 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8080757b0 0x7fa80819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8081100e0 0x7fa8081a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa808115c60 con 0x7fa808076100 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa808115ae0 con 0x7fa8080757b0 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa80f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa808115de0 con 0x7fa8081100e0 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa7ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa7ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80819ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38712/0 (socket says 172.21.15.39:38712) 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa807e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8080757b0 0x7fa80819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.723+0000 7fa7ffe00640 1 -- 172.21.15.39:0/1685798665 learned_addr learned my addr 172.21.15.39:0/1685798665 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa80d200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8081100e0 0x7fa8081a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa807e00640 1 -- 172.21.15.39:0/1685798665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8081100e0 msgr2=0x7fa8081a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa807e00640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8081100e0 0x7fa8081a1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa807e00640 1 -- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 msgr2=0x7fa80819ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa807e00640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80819ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa807e00640 1 -- 172.21.15.39:0/1685798665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8081a26a0 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa807e00640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8080757b0 0x7fa80819c730 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800e9f0 tx=0x7fa7f800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800ccc0 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8081a2990 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7f80108e0 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.724+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8018840 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.725+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8081a90f0 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.725+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa808076cc0 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.726+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fa7f8010430 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.726+0000 7fa805600640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7ec03fb70 0x7fa7ec042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.726+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa7f8054910 con 0x7fa8080757b0 2024-09-21T00:53:25.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.726+0000 7fa7ffe00640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7ec03fb70 0x7fa7ec042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.727+0000 7fa7ffe00640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7ec03fb70 0x7fa7ec042080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0005f70 tx=0x7fa7f0009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.731+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa7f8014230 con 0x7fa8080757b0 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.931+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fa8081a93e0 con 0x7fa8080757b0 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.941+0000 7fa805600640 1 -- 172.21.15.39:0/1685798665 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v22)=0 v22) ==== 119+0+0 (secure 0 0 0) 0x7fa7f80189e0 con 0x7fa8080757b0 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.943+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7ec03fb70 msgr2=0x7fa7ec042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.943+0000 7fa80f200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7ec03fb70 0x7fa7ec042080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0005f70 tx=0x7fa7f0009560 comp rx=0 tx=0).stop 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.943+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8080757b0 msgr2=0x7fa80819c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.943+0000 7fa80f200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8080757b0 0x7fa80819c730 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800e9f0 tx=0x7fa7f800edf0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 shutdown_connections 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8081100e0 0x7fa8081a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa808076100 0x7fa80819ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7ec03fb70 0x7fa7ec042080 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 --2- 172.21.15.39:0/1685798665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8080757b0 0x7fa80819c730 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:25.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 >> 172.21.15.39:0/1685798665 conn(0x7fa8080ff610 msgr2=0x7fa808109c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:25.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 shutdown_connections 2024-09-21T00:53:25.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:22.944+0000 7fa80f200640 1 -- 172.21.15.39:0/1685798665 wait complete. 2024-09-21T00:53:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:25 smithi039 ceph-mon[30415]: mgrmap e19: x(active, since 16s), standbys: z, y 2024-09-21T00:53:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T00:53:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:25 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1407835460' entity='client.admin' 2024-09-21T00:53:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:25 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/616833471' entity='client.admin' 2024-09-21T00:53:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:25 smithi150 ceph-mon[34945]: mgrmap e19: x(active, since 16s), standbys: z, y 2024-09-21T00:53:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T00:53:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:25 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1407835460' entity='client.admin' 2024-09-21T00:53:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:25 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/616833471' entity='client.admin' 2024-09-21T00:53:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:25 smithi160 ceph-mon[34696]: mgrmap e19: x(active, since 16s), standbys: z, y 2024-09-21T00:53:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T00:53:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:25 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1407835460' entity='client.admin' 2024-09-21T00:53:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:25 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/616833471' entity='client.admin' 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 -- 172.21.15.39:0/2174905254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c1094b0 msgr2=0x7f3b7c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 --2- 172.21.15.39:0/2174905254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c1094b0 0x7f3b7c1098c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009900 tx=0x7f3b6802a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 -- 172.21.15.39:0/2174905254 shutdown_connections 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 --2- 172.21.15.39:0/2174905254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c109e00 0x7f3b7c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 --2- 172.21.15.39:0/2174905254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c1094b0 0x7f3b7c1098c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 --2- 172.21.15.39:0/2174905254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b7c103b70 0x7f3b7c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 -- 172.21.15.39:0/2174905254 >> 172.21.15.39:0/2174905254 conn(0x7f3b7c0ff5e0 msgr2=0x7f3b7c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 -- 172.21.15.39:0/2174905254 shutdown_connections 2024-09-21T00:53:26.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.055+0000 7f3b84000640 1 -- 172.21.15.39:0/2174905254 wait complete. 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.057+0000 7f3b84000640 1 Processor -- start 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.057+0000 7f3b84000640 1 -- start start 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.057+0000 7f3b84000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 0x7f3b7c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b84000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b7c1094b0 0x7f3b7c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b84000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c109e00 0x7f3b7c1a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b84000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c113ac0 con 0x7f3b7c1094b0 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b84000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c113940 con 0x7f3b7c109e00 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b84000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c113c40 con 0x7f3b7c103b70 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 0x7f3b7c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 0x7f3b7c19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36438/0 (socket says 172.21.15.39:36438) 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 -- 172.21.15.39:0/3465751000 learned_addr learned my addr 172.21.15.39:0/3465751000 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b80c00640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b7c1094b0 0x7f3b7c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b82000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c109e00 0x7f3b7c1a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:26.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 -- 172.21.15.39:0/3465751000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c109e00 msgr2=0x7f3b7c1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c109e00 0x7f3b7c1a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 -- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b7c1094b0 msgr2=0x7f3b7c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b7c1094b0 0x7f3b7c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.058+0000 7f3b81600640 1 -- 172.21.15.39:0/3465751000 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c1a2620 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.059+0000 7f3b81600640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 0x7f3b7c19c6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000ea60 tx=0x7f3b7000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.059+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000cde0 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.059+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b700035f0 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.059+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7c1a2910 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.059+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70010690 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.059+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7c1a9070 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.061+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7c103f80 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.061+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f3b70010830 con 0x7f3b7c103b70 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.061+0000 7f3b7a000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b6403fb70 0x7f3b64042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:26.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.062+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f3b700549f0 con 0x7f3b7c103b70 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.062+0000 7f3b80c00640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b6403fb70 0x7f3b64042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.062+0000 7f3b80c00640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b6403fb70 0x7f3b64042080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009230 tx=0x7f3b68031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.066+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3b700173d0 con 0x7f3b7c103b70 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.278+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f3b7c1a9460 con 0x7f3b7c103b70 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.286+0000 7f3b7a000640 1 -- 172.21.15.39:0/3465751000 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v18)=0 v18) ==== 120+0+0 (secure 0 0 0) 0x7f3b700046c0 con 0x7f3b7c103b70 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b6403fb70 msgr2=0x7f3b64042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b6403fb70 0x7f3b64042080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009230 tx=0x7f3b68031400 comp rx=0 tx=0).stop 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 msgr2=0x7f3b7c19c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 0x7f3b7c19c6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000ea60 tx=0x7f3b7000ef20 comp rx=0 tx=0).stop 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 shutdown_connections 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b6403fb70 0x7f3b64042080 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b7c109e00 0x7f3b7c1a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b7c1094b0 0x7f3b7c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 --2- 172.21.15.39:0/3465751000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b7c103b70 0x7f3b7c19c6b0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000ea60 tx=0x7f3b7000ef20 comp rx=0 tx=0).stop 2024-09-21T00:53:26.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 >> 172.21.15.39:0/3465751000 conn(0x7f3b7c0ff5e0 msgr2=0x7f3b7c108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:26.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.290+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 shutdown_connections 2024-09-21T00:53:26.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:21.291+0000 7f3b84000640 1 -- 172.21.15.39:0/3465751000 wait complete. 2024-09-21T00:53:27.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 -- 172.21.15.39:0/2254926235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b381092d0 msgr2=0x7f5b381096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:27.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 --2- 172.21.15.39:0/2254926235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b381092d0 0x7f5b381096e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009900 tx=0x7f5b2402a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 -- 172.21.15.39:0/2254926235 shutdown_connections 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 --2- 172.21.15.39:0/2254926235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b38109c20 0x7f5b38110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 --2- 172.21.15.39:0/2254926235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b381092d0 0x7f5b381096e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 --2- 172.21.15.39:0/2254926235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 0x7f5b38103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 -- 172.21.15.39:0/2254926235 >> 172.21.15.39:0/2254926235 conn(0x7f5b380ff3e0 msgr2=0x7f5b38101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 -- 172.21.15.39:0/2254926235 shutdown_connections 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.376+0000 7f5b37e00640 1 -- 172.21.15.39:0/2254926235 wait complete. 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 Processor -- start 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 -- start start 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 0x7f5b3819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 0x7f5b3819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b38109c20 0x7f5b381a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381138e0 con 0x7f5b381092d0 2024-09-21T00:53:27.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b38113760 con 0x7f5b38103990 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.377+0000 7f5b37e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5b38113a60 con 0x7f5b38109c20 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 0x7f5b3819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 0x7f5b3819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 0x7f5b3819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:56628/0 (socket says 172.21.15.39:56628) 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 0x7f5b3819cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60682/0 (socket says 172.21.15.39:60682) 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36a00640 1 -- 172.21.15.39:0/24802730 learned_addr learned my addr 172.21.15.39:0/24802730 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b37400640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b38109c20 0x7f5b381a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 -- 172.21.15.39:0/24802730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b38109c20 msgr2=0x7f5b381a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b38109c20 0x7f5b381a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 -- 172.21.15.39:0/24802730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 msgr2=0x7f5b3819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 0x7f5b3819c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.378+0000 7f5b36000640 1 -- 172.21.15.39:0/24802730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b381a24a0 con 0x7f5b381092d0 2024-09-21T00:53:27.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.379+0000 7f5b36000640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 0x7f5b3819cad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5b2402a910 tx=0x7f5b24031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.379+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2403f070 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.379+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b24003aa0 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.379+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24035070 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.379+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b381a2730 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.379+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b381a8ef0 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.380+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b38104c10 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.382+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f5b24003c50 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.385+0000 7f5b2f400640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5b1c0b6550 0x7f5b1c0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.385+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f5b24105850 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.385+0000 7f5b36a00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5b1c0b6550 0x7f5b1c0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.386+0000 7f5b36a00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5b1c0b6550 0x7f5b1c0b8a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009c90 tx=0x7f5b28005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.389+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5b240cdc40 con 0x7f5b381092d0 2024-09-21T00:53:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.586+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f5b381a90c0 con 0x7f5b381092d0 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.590+0000 7f5b2f400640 1 -- 172.21.15.39:0/24802730 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v25)=0 v25) ==== 116+0+0 (secure 0 0 0) 0x7f5b240d2020 con 0x7f5b381092d0 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5b1c0b6550 msgr2=0x7f5b1c0b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5b1c0b6550 0x7f5b1c0b8a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009c90 tx=0x7f5b28005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 msgr2=0x7f5b3819cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 0x7f5b3819cad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5b2402a910 tx=0x7f5b24031a20 comp rx=0 tx=0).stop 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 shutdown_connections 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b38109c20 0x7f5b381a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b381092d0 0x7f5b3819cad0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5b1c0b6550 0x7f5b1c0b8a60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 --2- 172.21.15.39:0/24802730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b38103990 0x7f5b3819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 >> 172.21.15.39:0/24802730 conn(0x7f5b380ff3e0 msgr2=0x7f5b3810e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 shutdown_connections 2024-09-21T00:53:27.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:26.593+0000 7f5b37e00640 1 -- 172.21.15.39:0/24802730 wait complete. 2024-09-21T00:53:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:27 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/24802730' entity='client.admin' 2024-09-21T00:53:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:27 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/24802730' entity='client.admin' 2024-09-21T00:53:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:27 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/24802730' entity='client.admin' 2024-09-21T00:53:28.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:28 smithi039 ceph-mon[30415]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:28.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:28 smithi160 ceph-mon[34696]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:28 smithi150 ceph-mon[34945]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:29.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.661+0000 7f01f4400640 1 -- 172.21.15.39:0/2956515568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec077940 msgr2=0x7f01ec077d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:29.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.661+0000 7f01f4400640 1 --2- 172.21.15.39:0/2956515568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec077940 0x7f01ec077d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009900 tx=0x7f01e002a410 comp rx=0 tx=0).stop 2024-09-21T00:53:29.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 -- 172.21.15.39:0/2956515568 shutdown_connections 2024-09-21T00:53:29.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 --2- 172.21.15.39:0/2956515568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec114750 0x7f01ec116b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 --2- 172.21.15.39:0/2956515568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec078290 0x7f01ec114210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 --2- 172.21.15.39:0/2956515568 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec077940 0x7f01ec077d50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 -- 172.21.15.39:0/2956515568 >> 172.21.15.39:0/2956515568 conn(0x7f01ec101860 msgr2=0x7f01ec103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:29.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 -- 172.21.15.39:0/2956515568 shutdown_connections 2024-09-21T00:53:29.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.662+0000 7f01f4400640 1 -- 172.21.15.39:0/2956515568 wait complete. 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.663+0000 7f01f4400640 1 Processor -- start 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.663+0000 7f01f4400640 1 -- start start 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.663+0000 7f01f4400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 0x7f01ec1a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.663+0000 7f01f4400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec078290 0x7f01ec1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f4400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec114750 0x7f01ec1a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f4400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec11a0c0 con 0x7f01ec078290 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f4400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec119f40 con 0x7f01ec077940 2024-09-21T00:53:29.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f4400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec11a240 con 0x7f01ec114750 2024-09-21T00:53:29.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f1a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 0x7f01ec1a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:29.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f1a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 0x7f01ec1a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:56678/0 (socket says 172.21.15.39:56678) 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f1a00640 1 -- 172.21.15.39:0/2193897480 learned_addr learned my addr 172.21.15.39:0/2193897480 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f1000640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec078290 0x7f01ec1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f2400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec114750 0x7f01ec1a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.664+0000 7f01f2400640 1 -- 172.21.15.39:0/2193897480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 msgr2=0x7f01ec1a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f2400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 0x7f01ec1a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f2400640 1 -- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec078290 msgr2=0x7f01ec1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f2400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec078290 0x7f01ec1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f2400640 1 -- 172.21.15.39:0/2193897480 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ec1a6a40 con 0x7f01ec114750 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f1a00640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 0x7f01ec1a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f2400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec114750 0x7f01ec1a6310 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009fd0 tx=0x7f01e400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f1000640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec078290 0x7f01ec1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:29.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e400ce20 con 0x7f01ec114750 2024-09-21T00:53:29.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ec1a6d30 con 0x7f01ec114750 2024-09-21T00:53:29.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.665+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ec1ad490 con 0x7f01ec114750 2024-09-21T00:53:29.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.666+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01e4005020 con 0x7f01ec114750 2024-09-21T00:53:29.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.666+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e4010670 con 0x7f01ec114750 2024-09-21T00:53:29.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.667+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01cc0052c0 con 0x7f01ec114750 2024-09-21T00:53:29.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.671+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f01e4004a40 con 0x7f01ec114750 2024-09-21T00:53:29.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.674+0000 7f01ea000640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01d40b6550 0x7f01d40b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:29.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.674+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f01e40e5c10 con 0x7f01ec114750 2024-09-21T00:53:29.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.674+0000 7f01f1a00640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01d40b6550 0x7f01d40b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:29.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.674+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f01e4116790 con 0x7f01ec114750 2024-09-21T00:53:29.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.675+0000 7f01f1a00640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01d40b6550 0x7f01d40b8a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01e0031440 tx=0x7f01e00058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:29.825 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:29 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3630339550' entity='client.admin' 2024-09-21T00:53:29.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.914+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f01cc0055b0 con 0x7f01ec114750 2024-09-21T00:53:29.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.922+0000 7f01ea000640 1 -- 172.21.15.39:0/2193897480 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v28)=0 v28) ==== 125+0+0 (secure 0 0 0) 0x7f01e40ae000 con 0x7f01ec114750 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01d40b6550 msgr2=0x7f01d40b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01d40b6550 0x7f01d40b8a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01e0031440 tx=0x7f01e00058d0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec114750 msgr2=0x7f01ec1a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec114750 0x7f01ec1a6310 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009fd0 tx=0x7f01e400ef10 comp rx=0 tx=0).stop 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 shutdown_connections 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01ec114750 0x7f01ec1a6310 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01ec078290 0x7f01ec1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01d40b6550 0x7f01d40b8a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 --2- 172.21.15.39:0/2193897480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01ec077940 0x7f01ec1a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:29.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.925+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 >> 172.21.15.39:0/2193897480 conn(0x7f01ec101860 msgr2=0x7f01ec116830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:29.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.926+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 shutdown_connections 2024-09-21T00:53:29.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.926+0000 7f01f4400640 1 -- 172.21.15.39:0/2193897480 wait complete. 2024-09-21T00:53:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:29 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3630339550' entity='client.admin' 2024-09-21T00:53:30.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:29 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3630339550' entity='client.admin' 2024-09-21T00:53:30.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.226+0000 7f7ad3200640 1 -- 172.21.15.39:0/3985202194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 msgr2=0x7f7ad4110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:30.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.226+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3985202194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad4110470 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009ca0 tx=0x7f7acc02a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 -- 172.21.15.39:0/3985202194 shutdown_connections 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3985202194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad4110470 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3985202194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ad41092f0 0x7f7ad4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3985202194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41039b0 0x7f7ad4103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 -- 172.21.15.39:0/3985202194 >> 172.21.15.39:0/3985202194 conn(0x7f7ad40ff440 msgr2=0x7f7ad4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 -- 172.21.15.39:0/3985202194 shutdown_connections 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.227+0000 7f7ad3200640 1 -- 172.21.15.39:0/3985202194 wait complete. 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.228+0000 7f7ad3200640 1 Processor -- start 2024-09-21T00:53:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.228+0000 7f7ad3200640 1 -- start start 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ad41039b0 0x7f7ad419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 0x7f7ad419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4113900 con 0x7f7ad4109c40 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad3200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4113780 con 0x7f7ad41092f0 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad3200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4113a80 con 0x7f7ad41039b0 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad1400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 0x7f7ad419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad1400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 0x7f7ad419a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:56704/0 (socket says 172.21.15.39:56704) 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60762/0 (socket says 172.21.15.39:60762) 2024-09-21T00:53:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad1400640 1 -- 172.21.15.39:0/3925587114 learned_addr learned my addr 172.21.15.39:0/3925587114 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.229+0000 7f7ad1e00640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ad41039b0 0x7f7ad419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad1400640 1 -- 172.21.15.39:0/3925587114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ad41039b0 msgr2=0x7f7ad419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad1400640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ad41039b0 0x7f7ad419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad1400640 1 -- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 msgr2=0x7f7ad419fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad1400640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad419fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad1400640 1 -- 172.21.15.39:0/3925587114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad41a0200 con 0x7f7ad41092f0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad1400640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 0x7f7ad419a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e9c0 tx=0x7f7ac000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000ccc0 con 0x7f7ad41092f0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad41a04f0 con 0x7f7ad41092f0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.230+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ac00054d0 con 0x7f7ad41092f0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.231+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0010430 con 0x7f7ad41092f0 2024-09-21T00:53:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.231+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad41a6c50 con 0x7f7ad41092f0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.232+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad4103dc0 con 0x7f7ad41092f0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.234+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f7ac0005020 con 0x7f7ad41092f0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.235+0000 7f7acaa00640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ab80b64d0 0x7f7ab80b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.236+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7ac00e5b50 con 0x7f7ad41092f0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.236+0000 7f7ad1e00640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ab80b64d0 0x7f7ab80b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.236+0000 7f7ad1e00640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ab80b64d0 0x7f7ab80b89e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009bd0 tx=0x7f7ac4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.240+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7ac00adf40 con 0x7f7ad41092f0 2024-09-21T00:53:30.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.462+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f7ad402e640 con 0x7f7ad41092f0 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.470+0000 7f7acaa00640 1 -- 172.21.15.39:0/3925587114 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v29)=0 v29) ==== 129+0+0 (secure 0 0 0) 0x7f7ac00b2320 con 0x7f7ad41092f0 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ab80b64d0 msgr2=0x7f7ab80b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ab80b64d0 0x7f7ab80b89e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009bd0 tx=0x7f7ac4005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 msgr2=0x7f7ad419a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 0x7f7ad419a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e9c0 tx=0x7f7ac000ee80 comp rx=0 tx=0).stop 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 shutdown_connections 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ad4109c40 0x7f7ad419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ab80b64d0 0x7f7ab80b89e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad41092f0 0x7f7ad419a830 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 --2- 172.21.15.39:0/3925587114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7ad41039b0 0x7f7ad419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 >> 172.21.15.39:0/3925587114 conn(0x7f7ad40ff440 msgr2=0x7f7ad4101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 shutdown_connections 2024-09-21T00:53:30.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.472+0000 7f7ad3200640 1 -- 172.21.15.39:0/3925587114 wait complete. 2024-09-21T00:53:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:30 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:30 smithi150 ceph-mon[34945]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:30 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:30 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:30 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:30 smithi160 ceph-mon[34696]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:30 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:30 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:30 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:30 smithi039 ceph-mon[30415]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:31.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:30 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:30 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 -- 172.21.15.39:0/1735789775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 msgr2=0x7f20281098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 --2- 172.21.15.39:0/1735789775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f20281098b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2010009900 tx=0x7f201002a410 comp rx=0 tx=0).stop 2024-09-21T00:53:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 -- 172.21.15.39:0/1735789775 shutdown_connections 2024-09-21T00:53:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 --2- 172.21.15.39:0/1735789775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2028109df0 0x7f2028110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 --2- 172.21.15.39:0/1735789775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f20281098b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 --2- 172.21.15.39:0/1735789775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028103b60 0x7f2028103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 -- 172.21.15.39:0/1735789775 >> 172.21.15.39:0/1735789775 conn(0x7f202807b850 msgr2=0x7f202807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.397+0000 7f202d600640 1 -- 172.21.15.39:0/1735789775 shutdown_connections 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.398+0000 7f202d600640 1 -- 172.21.15.39:0/1735789775 wait complete. 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.399+0000 7f202d600640 1 Processor -- start 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.399+0000 7f202d600640 1 -- start start 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f202d600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2028103b60 0x7f2028075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f202d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f2028075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f202d600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028109df0 0x7f2028076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f202d600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2028113b00 con 0x7f20281094a0 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f202d600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2028113980 con 0x7f2028109df0 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f202d600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2028113c80 con 0x7f2028103b60 2024-09-21T00:53:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f201fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f2028075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f201fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f2028075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60690/0 (socket says 172.21.15.39:60690) 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f201fe00640 1 -- 172.21.15.39:0/3630339550 learned_addr learned my addr 172.21.15.39:0/3630339550 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f2027400640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028109df0 0x7f2028076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.400+0000 7f2026a00640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2028103b60 0x7f2028075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201fe00640 1 -- 172.21.15.39:0/3630339550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2028103b60 msgr2=0x7f2028075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201fe00640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2028103b60 0x7f2028075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201fe00640 1 -- 172.21.15.39:0/3630339550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028109df0 msgr2=0x7f2028076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201fe00640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028109df0 0x7f2028076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201fe00640 1 -- 172.21.15.39:0/3630339550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20281a8790 con 0x7f20281094a0 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f2027400640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028109df0 0x7f2028076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201fe00640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f2028075c50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2010005ab0 tx=0x7f2010003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201003f070 con 0x7f20281094a0 2024-09-21T00:53:31.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2010031b50 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010035030 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20281a8a20 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.401+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20281a8f50 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.402+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2028103f70 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.405+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f2010042070 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.406+0000 7f201f400640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f200c0b6550 0x7f200c0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.406+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2010105cf0 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.406+0000 7f2026a00640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f200c0b6550 0x7f200c0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.407+0000 7f2026a00640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f200c0b6550 0x7f200c0b8a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2028077160 tx=0x7f2018005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.410+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f20100d24c0 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.618+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f202802e640 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f201f400640 1 -- 172.21.15.39:0/3630339550 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v26)=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f20100d7400 con 0x7f20281094a0 2024-09-21T00:53:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f200c0b6550 msgr2=0x7f200c0b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f200c0b6550 0x7f200c0b8a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2028077160 tx=0x7f2018005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 msgr2=0x7f2028075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f2028075c50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2010005ab0 tx=0x7f2010003b90 comp rx=0 tx=0).stop 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 shutdown_connections 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f200c0b6550 0x7f200c0b8a60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2028109df0 0x7f2028076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281094a0 0x7f2028075c50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 --2- 172.21.15.39:0/3630339550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2028103b60 0x7f2028075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 >> 172.21.15.39:0/3630339550 conn(0x7f202807b850 msgr2=0x7f20281054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 shutdown_connections 2024-09-21T00:53:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:28.622+0000 7f202d600640 1 -- 172.21.15.39:0/3630339550 wait complete. 2024-09-21T00:53:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 -- 172.21.15.39:0/1652294254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09781049d0 msgr2=0x7f0978104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 --2- 172.21.15.39:0/1652294254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09781049d0 0x7f0978104de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f096c009900 tx=0x7f096c02a410 comp rx=0 tx=0).stop 2024-09-21T00:53:32.247 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:31 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/581695325' entity='client.admin' 2024-09-21T00:53:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 -- 172.21.15.39:0/1652294254 shutdown_connections 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 --2- 172.21.15.39:0/1652294254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 0x7f0978110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 --2- 172.21.15.39:0/1652294254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0978105320 0x7f097810d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 --2- 172.21.15.39:0/1652294254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09781049d0 0x7f0978104de0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 -- 172.21.15.39:0/1652294254 >> 172.21.15.39:0/1652294254 conn(0x7f0978079740 msgr2=0x7f0978079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.245+0000 7f0980400640 1 -- 172.21.15.39:0/1652294254 shutdown_connections 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.246+0000 7f0980400640 1 -- 172.21.15.39:0/1652294254 wait complete. 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.247+0000 7f0980400640 1 Processor -- start 2024-09-21T00:53:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.247+0000 7f0980400640 1 -- start start 2024-09-21T00:53:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f0980400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09781049d0 0x7f097819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f0980400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0978105320 0x7f097819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f0980400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 0x7f097819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f0980400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0978113ab0 con 0x7f0978105320 2024-09-21T00:53:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f0980400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0978113930 con 0x7f097810de30 2024-09-21T00:53:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f0980400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0978113c30 con 0x7f09781049d0 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f097e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 0x7f097819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f097e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 0x7f097819fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:56758/0 (socket says 172.21.15.39:56758) 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f097da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09781049d0 0x7f097819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f097e400640 1 -- 172.21.15.39:0/3433007839 learned_addr learned my addr 172.21.15.39:0/3433007839 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.248+0000 7f097d000640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0978105320 0x7f097819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097da00640 1 -- 172.21.15.39:0/3433007839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 msgr2=0x7f097819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097da00640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 0x7f097819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097da00640 1 -- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0978105320 msgr2=0x7f097819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097da00640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0978105320 0x7f097819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097da00640 1 -- 172.21.15.39:0/3433007839 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09781a0450 con 0x7f09781049d0 2024-09-21T00:53:32.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097d000640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0978105320 0x7f097819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:32.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.249+0000 7f097da00640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09781049d0 0x7f097819a4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f096c02a930 tx=0x7f096c031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:32.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.250+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c035050 con 0x7f09781049d0 2024-09-21T00:53:32.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.250+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f096c004410 con 0x7f09781049d0 2024-09-21T00:53:32.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.250+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09781a06e0 con 0x7f09781049d0 2024-09-21T00:53:32.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.250+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c003ba0 con 0x7f09781049d0 2024-09-21T00:53:32.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.250+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09781025f0 con 0x7f09781049d0 2024-09-21T00:53:32.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.251+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097806c240 con 0x7f09781049d0 2024-09-21T00:53:32.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.253+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f096c03a670 con 0x7f09781049d0 2024-09-21T00:53:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.255+0000 7f0976000640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09600b65d0 0x7f09600b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.255+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f096c1096b0 con 0x7f09781049d0 2024-09-21T00:53:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.255+0000 7f097d000640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09600b65d0 0x7f09600b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.256+0000 7f097d000640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09600b65d0 0x7f09600b8ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0978079ff0 tx=0x7f0964005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.257+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f096c0d1aa0 con 0x7f09781049d0 2024-09-21T00:53:32.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:31 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/581695325' entity='client.admin' 2024-09-21T00:53:32.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:31 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/581695325' entity='client.admin' 2024-09-21T00:53:32.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.456+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f0978104de0 con 0x7f09781049d0 2024-09-21T00:53:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.457+0000 7f0976000640 1 -- 172.21.15.39:0/3433007839 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v30)=0 v30) ==== 106+0+0 (secure 0 0 0) 0x7f096c0d5e80 con 0x7f09781049d0 2024-09-21T00:53:32.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.459+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09600b65d0 msgr2=0x7f09600b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09600b65d0 0x7f09600b8ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0978079ff0 tx=0x7f0964005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:32.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09781049d0 msgr2=0x7f097819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09781049d0 0x7f097819a4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f096c02a930 tx=0x7f096c031a20 comp rx=0 tx=0).stop 2024-09-21T00:53:32.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 shutdown_connections 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09600b65d0 0x7f09600b8ae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f097810de30 0x7f097819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0978105320 0x7f097819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 --2- 172.21.15.39:0/3433007839 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09781049d0 0x7f097819a4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 >> 172.21.15.39:0/3433007839 conn(0x7f0978079740 msgr2=0x7f09781007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 shutdown_connections 2024-09-21T00:53:32.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.460+0000 7f0980400640 1 -- 172.21.15.39:0/3433007839 wait complete. 2024-09-21T00:53:32.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 -- 172.21.15.39:0/2534679388 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea201094c0 msgr2=0x7fea201098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 --2- 172.21.15.39:0/2534679388 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea201094c0 0x7fea201098d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009860 tx=0x7fea0c02a340 comp rx=0 tx=0).stop 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 -- 172.21.15.39:0/2534679388 shutdown_connections 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 --2- 172.21.15.39:0/2534679388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea20109e10 0x7fea20110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 --2- 172.21.15.39:0/2534679388 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea201094c0 0x7fea201098d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 --2- 172.21.15.39:0/2534679388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea20103b80 0x7fea20103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 -- 172.21.15.39:0/2534679388 >> 172.21.15.39:0/2534679388 conn(0x7fea200ff590 msgr2=0x7fea201019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 -- 172.21.15.39:0/2534679388 shutdown_connections 2024-09-21T00:53:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.607+0000 7fea27a00640 1 -- 172.21.15.39:0/2534679388 wait complete. 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.608+0000 7fea27a00640 1 Processor -- start 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.608+0000 7fea27a00640 1 -- start start 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea27a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 0x7fea2019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea27a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea201094c0 0x7fea2019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea27a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea20109e10 0x7fea201a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea27a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fea20113ad0 con 0x7fea201094c0 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea27a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fea20113950 con 0x7fea20109e10 2024-09-21T00:53:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea27a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fea20113c50 con 0x7fea20103b80 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea25000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 0x7fea2019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea1fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea201094c0 0x7fea2019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea25a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea20109e10 0x7fea201a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea25000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 0x7fea2019c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34948/0 (socket says 172.21.15.39:34948) 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.609+0000 7fea25000640 1 -- 172.21.15.39:0/439790287 learned_addr learned my addr 172.21.15.39:0/439790287 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea25000640 1 -- 172.21.15.39:0/439790287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea20109e10 msgr2=0x7fea201a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea25000640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea20109e10 0x7fea201a1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea25000640 1 -- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea201094c0 msgr2=0x7fea2019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea25000640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea201094c0 0x7fea2019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea25000640 1 -- 172.21.15.39:0/439790287 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea201a2650 con 0x7fea20103b80 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea25000640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 0x7fea2019c6e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e9c0 tx=0x7fea1400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:32.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1400cde0 con 0x7fea20103b80 2024-09-21T00:53:32.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea201a2940 con 0x7fea20103b80 2024-09-21T00:53:32.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea140035f0 con 0x7fea20103b80 2024-09-21T00:53:32.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.610+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14010690 con 0x7fea20103b80 2024-09-21T00:53:32.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.611+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea201a90a0 con 0x7fea20103b80 2024-09-21T00:53:32.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.612+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea20103f90 con 0x7fea20103b80 2024-09-21T00:53:32.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.614+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fea14004000 con 0x7fea20103b80 2024-09-21T00:53:32.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.616+0000 7fea1d600640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea080b64d0 0x7fea080b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:32.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.616+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fea140e5ba0 con 0x7fea20103b80 2024-09-21T00:53:32.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.616+0000 7fea1fe00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea080b64d0 0x7fea080b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:32.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.617+0000 7fea1fe00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea080b64d0 0x7fea080b89e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fea0c035040 tx=0x7fea0c030340 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:32.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.617+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fea140ae020 con 0x7fea20103b80 2024-09-21T00:53:32.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.821+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7fea201a9470 con 0x7fea20103b80 2024-09-21T00:53:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.829+0000 7fea1d600640 1 -- 172.21.15.39:0/439790287 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v32)=0 v32) ==== 115+0+0 (secure 0 0 0) 0x7fea140b2400 con 0x7fea20103b80 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea080b64d0 msgr2=0x7fea080b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea080b64d0 0x7fea080b89e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fea0c035040 tx=0x7fea0c030340 comp rx=0 tx=0).stop 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 msgr2=0x7fea2019c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 0x7fea2019c6e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e9c0 tx=0x7fea1400ee80 comp rx=0 tx=0).stop 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 shutdown_connections 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea080b64d0 0x7fea080b89e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea20109e10 0x7fea201a1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea201094c0 0x7fea2019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 --2- 172.21.15.39:0/439790287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea20103b80 0x7fea2019c6e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:32.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 >> 172.21.15.39:0/439790287 conn(0x7fea200ff590 msgr2=0x7fea20107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:32.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.831+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 shutdown_connections 2024-09-21T00:53:32.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.832+0000 7fea27a00640 1 -- 172.21.15.39:0/439790287 wait complete. 2024-09-21T00:53:33.021 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:32 smithi039 ceph-mon[30415]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:33.021 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:32 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:33.021 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:32 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 -- 172.21.15.39:0/4063748844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 msgr2=0x7fb7901098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 --2- 172.21.15.39:0/4063748844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb7901098d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009860 tx=0x7fb77c02a340 comp rx=0 tx=0).stop 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 -- 172.21.15.39:0/4063748844 shutdown_connections 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 --2- 172.21.15.39:0/4063748844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb790109e10 0x7fb790110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 --2- 172.21.15.39:0/4063748844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb7901098d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 --2- 172.21.15.39:0/4063748844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb790103b80 0x7fb790103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 -- 172.21.15.39:0/4063748844 >> 172.21.15.39:0/4063748844 conn(0x7fb7900ff5d0 msgr2=0x7fb790101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 -- 172.21.15.39:0/4063748844 shutdown_connections 2024-09-21T00:53:33.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.201+0000 7fb797a00640 1 -- 172.21.15.39:0/4063748844 wait complete. 2024-09-21T00:53:33.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.202+0000 7fb797a00640 1 Processor -- start 2024-09-21T00:53:33.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.202+0000 7fb797a00640 1 -- start start 2024-09-21T00:53:33.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb797a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb790103b80 0x7fb790075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb797a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb790075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb797a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb790109e10 0x7fb790076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb797a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb790113ad0 con 0x7fb790109e10 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb797a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb790113950 con 0x7fb7901094c0 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb797a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb790113c50 con 0x7fb790103b80 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb795a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb790109e10 0x7fb790076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb78fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb790075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb78fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb790075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:40624/0 (socket says 172.21.15.39:40624) 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb78fe00640 1 -- 172.21.15.39:0/598407653 learned_addr learned my addr 172.21.15.39:0/598407653 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:33.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.203+0000 7fb795000640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb790103b80 0x7fb790075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78fe00640 1 -- 172.21.15.39:0/598407653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb790103b80 msgr2=0x7fb790075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78fe00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb790103b80 0x7fb790075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78fe00640 1 -- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb790109e10 msgr2=0x7fb790076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78fe00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb790109e10 0x7fb790076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78fe00640 1 -- 172.21.15.39:0/598407653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7901a87e0 con 0x7fb7901094c0 2024-09-21T00:53:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78fe00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb790075c50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb77c035040 tx=0x7fb77c032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:33.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.204+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c037040 con 0x7fb7901094c0 2024-09-21T00:53:33.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.205+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb77c030c10 con 0x7fb7901094c0 2024-09-21T00:53:33.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.205+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7901a8a70 con 0x7fb7901094c0 2024-09-21T00:53:33.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.205+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0035f0 con 0x7fb7901094c0 2024-09-21T00:53:33.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.205+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7901a8f40 con 0x7fb7901094c0 2024-09-21T00:53:33.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.206+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb790103f90 con 0x7fb7901094c0 2024-09-21T00:53:33.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.208+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fb77c043020 con 0x7fb7901094c0 2024-09-21T00:53:33.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.210+0000 7fb78d600640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7780b6550 0x7fb7780b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.211+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb77c104890 con 0x7fb7901094c0 2024-09-21T00:53:33.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.211+0000 7fb795000640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7780b6550 0x7fb7780b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.211+0000 7fb795000640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7780b6550 0x7fb7780b8a60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7840096c0 tx=0x7fb784009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:33.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.214+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb77c04a060 con 0x7fb7901094c0 2024-09-21T00:53:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 -- 172.21.15.39:0/740396024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc0757b0 msgr2=0x7f46cc075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 --2- 172.21.15.39:0/740396024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc0757b0 0x7f46cc075bc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009900 tx=0x7f46bc02a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 -- 172.21.15.39:0/740396024 shutdown_connections 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 --2- 172.21.15.39:0/740396024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc10ff10 0x7f46cc112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 --2- 172.21.15.39:0/740396024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 0x7f46cc10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 --2- 172.21.15.39:0/740396024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc0757b0 0x7f46cc075bc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 -- 172.21.15.39:0/740396024 >> 172.21.15.39:0/740396024 conn(0x7f46cc0ff400 msgr2=0x7f46cc101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 -- 172.21.15.39:0/740396024 shutdown_connections 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.553+0000 7f46d0a00640 1 -- 172.21.15.39:0/740396024 wait complete. 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.554+0000 7f46d0a00640 1 Processor -- start 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 -- start start 2024-09-21T00:53:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 0x7f46cc19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 0x7f46cc19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc10ff10 0x7f46cc1a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc115ac0 con 0x7f46cc10ff10 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc115940 con 0x7f46cc076100 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46d0a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc115c40 con 0x7f46cc0757b0 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.555+0000 7f46cb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 0x7f46cc19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 0x7f46cc19c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34836/0 (socket says 172.21.15.39:34836) 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 -- 172.21.15.39:0/1477397978 learned_addr learned my addr 172.21.15.39:0/1477397978 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cbe00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc10ff10 0x7f46cc1a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46caa00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 0x7f46cc19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 -- 172.21.15.39:0/1477397978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 msgr2=0x7f46cc19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 0x7f46cc19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 -- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc10ff10 msgr2=0x7f46cc1a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc10ff10 0x7f46cc1a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cb400640 1 -- 172.21.15.39:0/1477397978 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc1a2460 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46caa00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 0x7f46cc19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.556+0000 7f46cbe00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc10ff10 0x7f46cc1a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.557+0000 7f46cb400640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 0x7f46cc19c4f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005940 tx=0x7f46bc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.557+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc03f030 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.557+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc1a26f0 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.557+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46bc003cb0 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.557+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc035070 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.557+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1a8eb0 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.558+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc076cc0 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.561+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f46bc042030 con 0x7f46cc0757b0 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.562+0000 7f46c3e00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46b00b6550 0x7f46b00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.562+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f46bc106200 con 0x7f46cc0757b0 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.562+0000 7f46caa00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46b00b6550 0x7f46b00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.563+0000 7f46caa00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46b00b6550 0x7f46b00b8a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f46cc112270 tx=0x7f46b800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.564+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46bc0d29d0 con 0x7f46cc0757b0 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.777+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f46cc075bc0 con 0x7f46cc0757b0 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.786+0000 7f46c3e00640 1 -- 172.21.15.39:0/1477397978 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v27)=0 v27) ==== 108+0+0 (secure 0 0 0) 0x7f46bc0d7910 con 0x7f46cc0757b0 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46b00b6550 msgr2=0x7f46b00b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46b00b6550 0x7f46b00b8a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f46cc112270 tx=0x7f46b800b410 comp rx=0 tx=0).stop 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 msgr2=0x7f46cc19c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 0x7f46cc19c4f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005940 tx=0x7f46bc0042f0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 shutdown_connections 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46cc10ff10 0x7f46cc1a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46b00b6550 0x7f46b00b8a60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46cc076100 0x7f46cc19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 --2- 172.21.15.39:0/1477397978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f46cc0757b0 0x7f46cc19c4f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 >> 172.21.15.39:0/1477397978 conn(0x7f46cc0ff400 msgr2=0x7f46cc109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:33.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 shutdown_connections 2024-09-21T00:53:33.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:29.789+0000 7f46d0a00640 1 -- 172.21.15.39:0/1477397978 wait complete. 2024-09-21T00:53:33.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:32 smithi150 ceph-mon[34945]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:33.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:32 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:33.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:32 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:33.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:32 smithi160 ceph-mon[34696]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:33.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:32 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:33.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:32 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:33.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.415+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7fb79002e640 con 0x7fb7901094c0 2024-09-21T00:53:33.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.416+0000 7fb78d600640 1 -- 172.21.15.39:0/598407653 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v32)=0 v32) ==== 108+0+0 (secure 0 0 0) 0x7fb77c0ccc80 con 0x7fb7901094c0 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7780b6550 msgr2=0x7fb7780b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7780b6550 0x7fb7780b8a60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7840096c0 tx=0x7fb784009260 comp rx=0 tx=0).stop 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 msgr2=0x7fb790075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb790075c50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb77c035040 tx=0x7fb77c032d50 comp rx=0 tx=0).stop 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 shutdown_connections 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb790109e10 0x7fb790076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7780b6550 0x7fb7780b8a60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7901094c0 0x7fb790075c50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 --2- 172.21.15.39:0/598407653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb790103b80 0x7fb790075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:33.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 >> 172.21.15.39:0/598407653 conn(0x7fb7900ff5d0 msgr2=0x7fb7901019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:33.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 shutdown_connections 2024-09-21T00:53:33.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:33.419+0000 7fb797a00640 1 -- 172.21.15.39:0/598407653 wait complete. 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 -- 172.21.15.39:0/3074988347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 msgr2=0x7f64fc110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 --2- 172.21.15.39:0/3074988347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 0x7f64fc110620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64f4009900 tx=0x7f64f402a3f0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 -- 172.21.15.39:0/3074988347 shutdown_connections 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 --2- 172.21.15.39:0/3074988347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 0x7f64fc110620 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 --2- 172.21.15.39:0/3074988347 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc1094a0 0x7f64fc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 --2- 172.21.15.39:0/3074988347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64fc103b60 0x7f64fc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 -- 172.21.15.39:0/3074988347 >> 172.21.15.39:0/3074988347 conn(0x7f64fc07b850 msgr2=0x7f64fc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 -- 172.21.15.39:0/3074988347 shutdown_connections 2024-09-21T00:53:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.320+0000 7f6503e00640 1 -- 172.21.15.39:0/3074988347 wait complete. 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.321+0000 7f6503e00640 1 Processor -- start 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 -- start start 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 0x7f64fc19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64fc1094a0 0x7f64fc19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 0x7f64fc19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc113ab0 con 0x7f64fc109df0 2024-09-21T00:53:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc113930 con 0x7f64fc1094a0 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6503e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc113c30 con 0x7f64fc103b60 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6501400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 0x7f64fc19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6501400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 0x7f64fc19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48424/0 (socket says 172.21.15.39:48424) 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6501400640 1 -- 172.21.15.39:0/3513422972 learned_addr learned my addr 172.21.15.39:0/3513422972 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.322+0000 7f6501e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 0x7f64fc19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6500a00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64fc1094a0 0x7f64fc19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6501400640 1 -- 172.21.15.39:0/3513422972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64fc1094a0 msgr2=0x7f64fc19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6501400640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64fc1094a0 0x7f64fc19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6501400640 1 -- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 msgr2=0x7f64fc19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6501400640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 0x7f64fc19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6501400640 1 -- 172.21.15.39:0/3513422972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64fc1a0370 con 0x7f64fc103b60 2024-09-21T00:53:34.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f6501400640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 0x7f64fc19a400 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f64f000ebc0 tx=0x7f64f000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:34.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.323+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f0010630 con 0x7f64fc103b60 2024-09-21T00:53:34.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.324+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64f0010c80 con 0x7f64fc103b60 2024-09-21T00:53:34.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.324+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64fc1a0660 con 0x7f64fc103b60 2024-09-21T00:53:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.324+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f0018b60 con 0x7f64fc103b60 2024-09-21T00:53:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.324+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64fc1a6dc0 con 0x7f64fc103b60 2024-09-21T00:53:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.325+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64fc103f70 con 0x7f64fc103b60 2024-09-21T00:53:34.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.327+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f64f0018d00 con 0x7f64fc103b60 2024-09-21T00:53:34.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.329+0000 7f64fa000640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64e40b6550 0x7f64e40b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:34.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.329+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f64f00e5fc0 con 0x7f64fc103b60 2024-09-21T00:53:34.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.329+0000 7f6500a00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64e40b6550 0x7f64e40b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:34.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.329+0000 7f6500a00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64e40b6550 0x7f64e40b8a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f64fc109170 tx=0x7f64e800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:34.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.330+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f64f00b2790 con 0x7f64fc103b60 2024-09-21T00:53:34.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.531+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f64fc02e640 con 0x7f64fc103b60 2024-09-21T00:53:34.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.539+0000 7f64fa000640 1 -- 172.21.15.39:0/3513422972 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v33)=0 v33) ==== 138+0+0 (secure 0 0 0) 0x7f64f00b76d0 con 0x7f64fc103b60 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64e40b6550 msgr2=0x7f64e40b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64e40b6550 0x7f64e40b8a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f64fc109170 tx=0x7f64e800b390 comp rx=0 tx=0).stop 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 msgr2=0x7f64fc19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 0x7f64fc19a400 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f64f000ebc0 tx=0x7f64f000c6a0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 shutdown_connections 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64fc109df0 0x7f64fc19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64e40b6550 0x7f64e40b8a60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64fc1094a0 0x7f64fc19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 --2- 172.21.15.39:0/3513422972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64fc103b60 0x7f64fc19a400 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:34.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 >> 172.21.15.39:0/3513422972 conn(0x7f64fc07b850 msgr2=0x7f64fc107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:34.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 shutdown_connections 2024-09-21T00:53:34.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:34.542+0000 7f6503e00640 1 -- 172.21.15.39:0/3513422972 wait complete. 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1738430063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e010de30 msgr2=0x7fa0e0110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1738430063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e010de30 0x7fa0e0110320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009ca0 tx=0x7fa0d802a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1738430063 shutdown_connections 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1738430063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e010de30 0x7fa0e0110320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1738430063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0e0105320 0x7fa0e010d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1738430063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e01049d0 0x7fa0e0104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1738430063 >> 172.21.15.39:0/1738430063 conn(0x7fa0e0079740 msgr2=0x7fa0e0079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1738430063 shutdown_connections 2024-09-21T00:53:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.074+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1738430063 wait complete. 2024-09-21T00:53:35.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.075+0000 7fa0e5e00640 1 Processor -- start 2024-09-21T00:53:35.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.075+0000 7fa0e5e00640 1 -- start start 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0e5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0e01049d0 0x7fa0e019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0e5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 0x7fa0e019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0e5e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e010de30 0x7fa0e01a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0e5e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e0113b00 con 0x7fa0e0105320 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0e5e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e0113980 con 0x7fa0e010de30 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0e5e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e0113c80 con 0x7fa0e01049d0 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 0x7fa0e019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0df400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e010de30 0x7fa0e01a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 0x7fa0e019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44512/0 (socket says 172.21.15.39:44512) 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0d7e00640 1 -- 172.21.15.39:0/1239977809 learned_addr learned my addr 172.21.15.39:0/1239977809 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:35.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.076+0000 7fa0dea00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0e01049d0 0x7fa0e019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:35.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0d7e00640 1 -- 172.21.15.39:0/1239977809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0e01049d0 msgr2=0x7fa0e019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:35.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0d7e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0e01049d0 0x7fa0e019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0d7e00640 1 -- 172.21.15.39:0/1239977809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e010de30 msgr2=0x7fa0e01a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:35.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0d7e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e010de30 0x7fa0e01a1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0d7e00640 1 -- 172.21.15.39:0/1239977809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e01a26a0 con 0x7fa0e0105320 2024-09-21T00:53:35.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0df400640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e010de30 0x7fa0e01a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0d7e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 0x7fa0e019ccd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa0c800e9c0 tx=0x7fa0c800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c800ccc0 con 0x7fa0e0105320 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c80054d0 con 0x7fa0e0105320 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8010430 con 0x7fa0e0105320 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.077+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e01a2990 con 0x7fa0e0105320 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.078+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e01a90f0 con 0x7fa0e0105320 2024-09-21T00:53:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.079+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e006c240 con 0x7fa0e0105320 2024-09-21T00:53:35.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.081+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fa0c8005000 con 0x7fa0e0105320 2024-09-21T00:53:35.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.083+0000 7fa0e4a00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0c40b64d0 0x7fa0c40b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:35.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.083+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa0c80e5c80 con 0x7fa0e0105320 2024-09-21T00:53:35.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.083+0000 7fa0dea00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0c40b64d0 0x7fa0c40b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:35.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.084+0000 7fa0dea00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0c40b64d0 0x7fa0c40b89e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0e006c940 tx=0x7fa0d0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:35.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.084+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa0c8056710 con 0x7fa0e0105320 2024-09-21T00:53:35.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.289+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fa0e0104de0 con 0x7fa0e0105320 2024-09-21T00:53:35.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.295+0000 7fa0e4a00640 1 -- 172.21.15.39:0/1239977809 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v34)=0 v34) ==== 111+0+0 (secure 0 0 0) 0x7fa0c80052f0 con 0x7fa0e0105320 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.297+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0c40b64d0 msgr2=0x7fa0c40b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.297+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0c40b64d0 0x7fa0c40b89e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0e006c940 tx=0x7fa0d0005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.297+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 msgr2=0x7fa0e019ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.297+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 0x7fa0e019ccd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa0c800e9c0 tx=0x7fa0c800ee80 comp rx=0 tx=0).stop 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 shutdown_connections 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0c40b64d0 0x7fa0c40b89e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0e010de30 0x7fa0e01a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0e0105320 0x7fa0e019ccd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 --2- 172.21.15.39:0/1239977809 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa0e01049d0 0x7fa0e019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:35.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 >> 172.21.15.39:0/1239977809 conn(0x7fa0e0079740 msgr2=0x7fa0e01036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:35.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 shutdown_connections 2024-09-21T00:53:35.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:35.298+0000 7fa0e5e00640 1 -- 172.21.15.39:0/1239977809 wait complete. 2024-09-21T00:53:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:35 smithi039 ceph-mon[30415]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:35 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:35 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1239977809' entity='client.admin' 2024-09-21T00:53:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:35 smithi150 ceph-mon[34945]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:35 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:35 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1239977809' entity='client.admin' 2024-09-21T00:53:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:35 smithi160 ceph-mon[34696]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:35 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:35 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1239977809' entity='client.admin' 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 -- 172.21.15.39:0/1154280748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 msgr2=0x7fc3c0103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/1154280748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c0103dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009900 tx=0x7fc3b402a410 comp rx=0 tx=0).stop 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 -- 172.21.15.39:0/1154280748 shutdown_connections 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/1154280748 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3c0109c40 0x7fc3c0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/1154280748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 0x7fc3c0109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/1154280748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c0103dc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 -- 172.21.15.39:0/1154280748 >> 172.21.15.39:0/1154280748 conn(0x7fc3c00ff400 msgr2=0x7fc3c0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:36.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 -- 172.21.15.39:0/1154280748 shutdown_connections 2024-09-21T00:53:36.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.552+0000 7fc3c6e00640 1 -- 172.21.15.39:0/1154280748 wait complete. 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.553+0000 7fc3c6e00640 1 Processor -- start 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.553+0000 7fc3c6e00640 1 -- start start 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 0x7fc3c019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3c0109c40 0x7fc3c01a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0113950 con 0x7fc3c01039b0 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01137d0 con 0x7fc3c01092f0 2024-09-21T00:53:36.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0113ad0 con 0x7fc3c0109c40 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c019c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44528/0 (socket says 172.21.15.39:44528) 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c5a00640 1 -- 172.21.15.39:0/4117991978 learned_addr learned my addr 172.21.15.39:0/4117991978 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c5000640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 0x7fc3c019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.554+0000 7fc3c6400640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3c0109c40 0x7fc3c01a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5000640 1 -- 172.21.15.39:0/4117991978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3c0109c40 msgr2=0x7fc3c01a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5000640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3c0109c40 0x7fc3c01a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5000640 1 -- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 msgr2=0x7fc3c019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5000640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5000640 1 -- 172.21.15.39:0/4117991978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c01a24b0 con 0x7fc3c01092f0 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5a00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:36.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c5000640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 0x7fc3c019cae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3c006b930 tx=0x7fc3ac00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:36.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac00cc40 con 0x7fc3c01092f0 2024-09-21T00:53:36.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ac00cde0 con 0x7fc3c01092f0 2024-09-21T00:53:36.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.555+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c01a27a0 con 0x7fc3c01092f0 2024-09-21T00:53:36.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.556+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac018780 con 0x7fc3c01092f0 2024-09-21T00:53:36.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.556+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c01a8f00 con 0x7fc3c01092f0 2024-09-21T00:53:36.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.557+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c0103dc0 con 0x7fc3c01092f0 2024-09-21T00:53:36.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.559+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fc3ac010a90 con 0x7fc3c01092f0 2024-09-21T00:53:36.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.560+0000 7fc3be000640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3a80b6550 0x7fc3a80b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:36.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.561+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc3ac0e59d0 con 0x7fc3c01092f0 2024-09-21T00:53:36.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.561+0000 7fc3c5a00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3a80b6550 0x7fc3a80b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:36.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.561+0000 7fc3c5a00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3a80b6550 0x7fc3a80b8a60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40096c0 tx=0x7fc3b4005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:36.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.565+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc3ac0b21a0 con 0x7fc3c01092f0 2024-09-21T00:53:36.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.769+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fc3c01a91e0 con 0x7fc3c01092f0 2024-09-21T00:53:36.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.778+0000 7fc3be000640 1 -- 172.21.15.39:0/4117991978 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v35)=0 v35) ==== 121+0+0 (secure 0 0 0) 0x7fc3ac0b70e0 con 0x7fc3c01092f0 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3a80b6550 msgr2=0x7fc3a80b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3a80b6550 0x7fc3a80b8a60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40096c0 tx=0x7fc3b4005a20 comp rx=0 tx=0).stop 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 msgr2=0x7fc3c019cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 0x7fc3c019cae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3c006b930 tx=0x7fc3ac00ee50 comp rx=0 tx=0).stop 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 shutdown_connections 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3c0109c40 0x7fc3c01a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3a80b6550 0x7fc3a80b8a60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3c01092f0 0x7fc3c019cae0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 --2- 172.21.15.39:0/4117991978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3c01039b0 0x7fc3c019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:36.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 >> 172.21.15.39:0/4117991978 conn(0x7fc3c00ff400 msgr2=0x7fc3c01089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:36.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 shutdown_connections 2024-09-21T00:53:36.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:36.781+0000 7fc3c6e00640 1 -- 172.21.15.39:0/4117991978 wait complete. 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 -- 172.21.15.39:0/3072645638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac110140 msgr2=0x7fd1ac112560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 --2- 172.21.15.39:0/3072645638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac110140 0x7fd1ac112560 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009ca0 tx=0x7fd1a402a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 -- 172.21.15.39:0/3072645638 shutdown_connections 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 --2- 172.21.15.39:0/3072645638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac110140 0x7fd1ac112560 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 --2- 172.21.15.39:0/3072645638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac076080 0x7fd1ac10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 --2- 172.21.15.39:0/3072645638 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 -- 172.21.15.39:0/3072645638 >> 172.21.15.39:0/3072645638 conn(0x7fd1ac07b850 msgr2=0x7fd1ac07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 -- 172.21.15.39:0/3072645638 shutdown_connections 2024-09-21T00:53:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.405+0000 7fd1b3400640 1 -- 172.21.15.39:0/3072645638 wait complete. 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.406+0000 7fd1b3400640 1 Processor -- start 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 -- start start 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac076080 0x7fd1ac19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac110140 0x7fd1ac1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac115cd0 con 0x7fd1ac076080 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac115b50 con 0x7fd1ac110140 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b3400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac115e50 con 0x7fd1ac075730 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1abe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac076080 0x7fd1ac19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac19c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48484/0 (socket says 172.21.15.39:48484) 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.407+0000 7fd1b0a00640 1 -- 172.21.15.39:0/2388653812 learned_addr learned my addr 172.21.15.39:0/2388653812 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b1400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac110140 0x7fd1ac1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b0a00640 1 -- 172.21.15.39:0/2388653812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac110140 msgr2=0x7fd1ac1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b0a00640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac110140 0x7fd1ac1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b0a00640 1 -- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac076080 msgr2=0x7fd1ac19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b0a00640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac076080 0x7fd1ac19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b0a00640 1 -- 172.21.15.39:0/2388653812 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac1a2590 con 0x7fd1ac075730 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b1400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac110140 0x7fd1ac1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1abe00640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac076080 0x7fd1ac19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b0a00640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac19c620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e9f0 tx=0x7fd1a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a000cd80 con 0x7fd1ac075730 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1a00108e0 con 0x7fd1ac075730 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.408+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ac1a2880 con 0x7fd1ac075730 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.409+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0018980 con 0x7fd1ac075730 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.409+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ac1a8fe0 con 0x7fd1ac075730 2024-09-21T00:53:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.410+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1ac075b40 con 0x7fd1ac075730 2024-09-21T00:53:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.412+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fd1a0010430 con 0x7fd1ac075730 2024-09-21T00:53:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.413+0000 7fd1a9600640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd1940b6550 0x7fd1940b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.413+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd1a00e5fe0 con 0x7fd1ac075730 2024-09-21T00:53:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.413+0000 7fd1abe00640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd1940b6550 0x7fd1940b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.414+0000 7fd1abe00640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd1940b6550 0x7fd1940b8a60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac111f20 tx=0x7fd198005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.415+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd1a00ae410 con 0x7fd1ac075730 2024-09-21T00:53:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.621+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fd1ac19d750 con 0x7fd1ac075730 2024-09-21T00:53:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.628+0000 7fd1a9600640 1 -- 172.21.15.39:0/2388653812 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v36)=0 v36) ==== 119+0+0 (secure 0 0 0) 0x7fd1a00b27f0 con 0x7fd1ac075730 2024-09-21T00:53:37.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd1940b6550 msgr2=0x7fd1940b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd1940b6550 0x7fd1940b8a60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac111f20 tx=0x7fd198005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 msgr2=0x7fd1ac19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac19c620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e9f0 tx=0x7fd1a000eeb0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 shutdown_connections 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd1940b6550 0x7fd1940b8a60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1ac110140 0x7fd1ac1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1ac076080 0x7fd1ac19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 --2- 172.21.15.39:0/2388653812 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd1ac075730 0x7fd1ac19c620 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 >> 172.21.15.39:0/2388653812 conn(0x7fd1ac07b850 msgr2=0x7fd1ac107ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 shutdown_connections 2024-09-21T00:53:37.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:37.631+0000 7fd1b3400640 1 -- 172.21.15.39:0/2388653812 wait complete. 2024-09-21T00:53:37.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 -- 172.21.15.39:0/1194120984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 msgr2=0x7fdc04110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 --2- 172.21.15.39:0/1194120984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc04110630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009ca0 tx=0x7fdbfc02a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 -- 172.21.15.39:0/1194120984 shutdown_connections 2024-09-21T00:53:37.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 --2- 172.21.15.39:0/1194120984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc04110630 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 --2- 172.21.15.39:0/1194120984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 0x7fdc041098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 --2- 172.21.15.39:0/1194120984 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 0x7fdc04103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 -- 172.21.15.39:0/1194120984 >> 172.21.15.39:0/1194120984 conn(0x7fdc040ff5e0 msgr2=0x7fdc04101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 -- 172.21.15.39:0/1194120984 shutdown_connections 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.508+0000 7fdc0a200640 1 -- 172.21.15.39:0/1194120984 wait complete. 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.509+0000 7fdc0a200640 1 Processor -- start 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 -- start start 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 0x7fdc0419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 0x7fdc0419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc0419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdc04113b10 con 0x7fdc04109e00 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdc04113990 con 0x7fdc041094b0 2024-09-21T00:53:37.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.510+0000 7fdc0a200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdc04113c90 con 0x7fdc04103b70 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.511+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc0419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.511+0000 7fdc03400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 0x7fdc0419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.511+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc0419fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60806/0 (socket says 172.21.15.39:60806) 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.511+0000 7fdc03e00640 1 -- 172.21.15.39:0/3266516513 learned_addr learned my addr 172.21.15.39:0/3266516513 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.511+0000 7fdc02a00640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 0x7fdc0419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.511+0000 7fdc03400640 1 -- 172.21.15.39:0/3266516513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 msgr2=0x7fdc0419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc03400640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 0x7fdc0419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc03400640 1 -- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 msgr2=0x7fdc0419fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc03400640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc0419fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc03400640 1 -- 172.21.15.39:0/3266516513 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc041a01c0 con 0x7fdc04103b70 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc02a00640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 0x7fdc0419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc03400640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 0x7fdc0419a440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400e9f0 tx=0x7fdbf400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400ccc0 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbf40108e0 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc041a04b0 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4018880 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.512+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc041a0a40 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.515+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc04103f80 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.518+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fdbf4010430 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.521+0000 7fdbfbe00640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdbe80b65d0 0x7fdbe80b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.521+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fdbf40e5aa0 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.521+0000 7fdc02a00640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdbe80b65d0 0x7fdbe80b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.522+0000 7fdc02a00640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdbe80b65d0 0x7fdbe80b8ae0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc04102dc0 tx=0x7fdbf0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.523+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdbf40b2270 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.728+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fdc0410bc50 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.735+0000 7fdbfbe00640 1 -- 172.21.15.39:0/3266516513 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v31)=0 v31) ==== 121+0+0 (secure 0 0 0) 0x7fdbf40b71b0 con 0x7fdc04103b70 2024-09-21T00:53:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdbe80b65d0 msgr2=0x7fdbe80b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdbe80b65d0 0x7fdbe80b8ae0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc04102dc0 tx=0x7fdbf0005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 msgr2=0x7fdc0419a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 0x7fdc0419a440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400e9f0 tx=0x7fdbf400eeb0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 shutdown_connections 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc04109e00 0x7fdc0419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdbe80b65d0 0x7fdbe80b8ae0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc041094b0 0x7fdc0419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 --2- 172.21.15.39:0/3266516513 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdc04103b70 0x7fdc0419a440 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:37.738 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:37 smithi039 ceph-mon[30415]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:37.738 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:37 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:37.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 >> 172.21.15.39:0/3266516513 conn(0x7fdc040ff5e0 msgr2=0x7fdc04106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:37.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 shutdown_connections 2024-09-21T00:53:37.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:32.738+0000 7fdc0a200640 1 -- 172.21.15.39:0/3266516513 wait complete. 2024-09-21T00:53:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:37 smithi150 ceph-mon[34945]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:37 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:37.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:37 smithi160 ceph-mon[34696]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:37.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:37 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:38.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:38 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' 2024-09-21T00:53:38.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:38 smithi039 ceph-mon[30415]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 -- 172.21.15.39:0/4235143389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7410109c20 msgr2=0x7f7410110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 --2- 172.21.15.39:0/4235143389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7410109c20 0x7f7410110450 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7408009ca0 tx=0x7f740802a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 -- 172.21.15.39:0/4235143389 shutdown_connections 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 --2- 172.21.15.39:0/4235143389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7410109c20 0x7f7410110450 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 --2- 172.21.15.39:0/4235143389 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f74101092d0 0x7f74101096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 --2- 172.21.15.39:0/4235143389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7410103990 0x7f7410103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 -- 172.21.15.39:0/4235143389 >> 172.21.15.39:0/4235143389 conn(0x7f74100ff440 msgr2=0x7f7410101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 -- 172.21.15.39:0/4235143389 shutdown_connections 2024-09-21T00:53:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.748+0000 7f740f200640 1 -- 172.21.15.39:0/4235143389 wait complete. 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.750+0000 7f740f200640 1 Processor -- start 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.750+0000 7f740f200640 1 -- start start 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7410103990 0x7f741019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 0x7f741019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7410109c20 0x7f741019fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f74101138e0 con 0x7f74101092d0 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7410113760 con 0x7f7410109c20 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7410113a60 con 0x7f7410103990 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 0x7f741019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 0x7f741019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60784/0 (socket says 172.21.15.39:60784) 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 -- 172.21.15.39:0/581695325 learned_addr learned my addr 172.21.15.39:0/581695325 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740de00640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7410103990 0x7f741019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:38.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 -- 172.21.15.39:0/581695325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7410103990 msgr2=0x7f741019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7410103990 0x7f741019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 -- 172.21.15.39:0/581695325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7410109c20 msgr2=0x7f741019fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.751+0000 7f740d400640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7410109c20 0x7f741019fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.752+0000 7f740d400640 1 -- 172.21.15.39:0/581695325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74101a0200 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.752+0000 7f740d400640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 0x7f741019a830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00d950 tx=0x7f73fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.752+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc017070 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.752+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73fc009c30 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.752+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74101a04f0 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.752+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc0047a0 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.753+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74101a6c50 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.754+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7410103da0 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.756+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f73fc004020 con 0x7f74101092d0 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.758+0000 7f7406a00640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73f40b64d0 0x7f73f40b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.758+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f73fc0e5810 con 0x7f74101092d0 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.758+0000 7f740de00640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73f40b64d0 0x7f73f40b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.759+0000 7f740de00640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73f40b64d0 0x7f73f40b89e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7400005d00 tx=0x7f740000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.760+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f73fc020020 con 0x7f74101092d0 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.964+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7f741002e640 con 0x7f74101092d0 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.970+0000 7f7406a00640 1 -- 172.21.15.39:0/581695325 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v30)=0 v30) ==== 115+0+0 (secure 0 0 0) 0x7f73fc0adc00 con 0x7f74101092d0 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73f40b64d0 msgr2=0x7f73f40b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73f40b64d0 0x7f73f40b89e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7400005d00 tx=0x7f740000b410 comp rx=0 tx=0).stop 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 msgr2=0x7f741019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 0x7f741019a830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00d950 tx=0x7f73fc00de10 comp rx=0 tx=0).stop 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 shutdown_connections 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73f40b64d0 0x7f73f40b89e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7410109c20 0x7f741019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74101092d0 0x7f741019a830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 --2- 172.21.15.39:0/581695325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7410103990 0x7f741019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.972+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 >> 172.21.15.39:0/581695325 conn(0x7f74100ff440 msgr2=0x7f7410108930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.974+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 shutdown_connections 2024-09-21T00:53:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:30.974+0000 7f740f200640 1 -- 172.21.15.39:0/581695325 wait complete. 2024-09-21T00:53:39.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:38 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' 2024-09-21T00:53:39.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:38 smithi150 ceph-mon[34945]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:38 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' 2024-09-21T00:53:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:38 smithi160 ceph-mon[34696]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:41.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:40 smithi160 ceph-mon[34696]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:41.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:40 smithi039 ceph-mon[30415]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:40 smithi150 ceph-mon[34945]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:43.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:42 smithi160 ceph-mon[34696]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:43.120 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:42 smithi039 ceph-mon[30415]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:42 smithi150 ceph-mon[34945]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:44.974 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:44 smithi039 ceph-mon[30415]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:44.976 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi039') 2024-09-21T00:53:44.976 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi039') 2024-09-21T00:53:44.976 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi039') 2024-09-21T00:53:44.976 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi039') 2024-09-21T00:53:44.977 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi039') 2024-09-21T00:53:44.977 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi039') 2024-09-21T00:53:44.977 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi039') 2024-09-21T00:53:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:44 smithi150 ceph-mon[34945]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:45.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:44 smithi160 ceph-mon[34696]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:45.416 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi039') 2024-09-21T00:53:47.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:46 smithi039 ceph-mon[30415]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:47.159 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:46 smithi160 ceph-mon[34696]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:47.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:46 smithi150 ceph-mon[34945]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:47.416 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:49.130 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:48 smithi160 ceph-mon[34696]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:49.130 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:49.130 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:49.130 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:49.130 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:49.131 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:49.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:48 smithi039 ceph-mon[30415]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:49.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:49.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:49.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:49.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:49.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 -- 172.21.15.39:0/3932961859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe01010c010 msgr2=0x7fe010112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 --2- 172.21.15.39:0/3932961859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe01010c010 0x7fe010112840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe008009ca0 tx=0x7fe00802a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 -- 172.21.15.39:0/3932961859 shutdown_connections 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 --2- 172.21.15.39:0/3932961859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe01010c010 0x7fe010112840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 --2- 172.21.15.39:0/3932961859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01010b6c0 0x7fe01010bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 --2- 172.21.15.39:0/3932961859 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe010105d80 0x7fe010106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 -- 172.21.15.39:0/3932961859 >> 172.21.15.39:0/3932961859 conn(0x7fe0101017d0 msgr2=0x7fe010103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:49.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 -- 172.21.15.39:0/3932961859 shutdown_connections 2024-09-21T00:53:49.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.201+0000 7fe017c00640 1 -- 172.21.15.39:0/3932961859 wait complete. 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.202+0000 7fe017c00640 1 Processor -- start 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.202+0000 7fe017c00640 1 -- start start 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe017c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 0x7fe01019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe017c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01010b6c0 0x7fe01019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe017c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe01010c010 0x7fe0101a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe017c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe010115d20 con 0x7fe010105d80 2024-09-21T00:53:49.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe017c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe010115ba0 con 0x7fe01010b6c0 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe017c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe010115ea0 con 0x7fe01010c010 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe015200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 0x7fe01019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe015200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 0x7fe01019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43298/0 (socket says 172.21.15.39:43298) 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe015200640 1 -- 172.21.15.39:0/727596018 learned_addr learned my addr 172.21.15.39:0/727596018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe015c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe01010c010 0x7fe0101a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.203+0000 7fe007e00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01010b6c0 0x7fe01019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe015200640 1 -- 172.21.15.39:0/727596018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe01010c010 msgr2=0x7fe0101a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe015200640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe01010c010 0x7fe0101a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe015200640 1 -- 172.21.15.39:0/727596018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01010b6c0 msgr2=0x7fe01019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:49.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe015200640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01010b6c0 0x7fe01019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe015200640 1 -- 172.21.15.39:0/727596018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0101a2660 con 0x7fe010105d80 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe015200640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 0x7fe01019c6f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe00000e9f0 tx=0x7fe00000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00000cd80 con 0x7fe010105d80 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0000054d0 con 0x7fe010105d80 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe000010690 con 0x7fe010105d80 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.204+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0101a2950 con 0x7fe010105d80 2024-09-21T00:53:49.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.205+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0101a90b0 con 0x7fe010105d80 2024-09-21T00:53:49.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.206+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe010106190 con 0x7fe010105d80 2024-09-21T00:53:49.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.208+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fe000004e30 con 0x7fe010105d80 2024-09-21T00:53:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.210+0000 7fe00e000640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdff40b64d0 0x7fdff40b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.210+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fe0000e5d80 con 0x7fe010105d80 2024-09-21T00:53:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.210+0000 7fe007e00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdff40b64d0 0x7fdff40b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.211+0000 7fe007e00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdff40b64d0 0x7fdff40b89e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe010104ed0 tx=0x7fdff8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.211+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe0000ae170 con 0x7fe010105d80 2024-09-21T00:53:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:48 smithi150 ceph-mon[34945]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T00:53:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T00:53:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:49.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.409+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7fe01019d820 con 0x7fe010105d80 2024-09-21T00:53:49.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.418+0000 7fe00e000640 1 -- 172.21.15.39:0/727596018 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v37)=0 v37) ==== 122+0+0 (secure 0 0 0) 0x7fe0000b2550 con 0x7fe010105d80 2024-09-21T00:53:49.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdff40b64d0 msgr2=0x7fdff40b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdff40b64d0 0x7fdff40b89e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe010104ed0 tx=0x7fdff8005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 msgr2=0x7fe01019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 0x7fe01019c6f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe00000e9f0 tx=0x7fe00000edf0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 shutdown_connections 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe01010c010 0x7fe0101a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdff40b64d0 0x7fdff40b89e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01010b6c0 0x7fe01019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 --2- 172.21.15.39:0/727596018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe010105d80 0x7fe01019c6f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 >> 172.21.15.39:0/727596018 conn(0x7fe0101017d0 msgr2=0x7fe01010ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 shutdown_connections 2024-09-21T00:53:49.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:49.421+0000 7fe017c00640 1 -- 172.21.15.39:0/727596018 wait complete. 2024-09-21T00:53:50.168 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.168 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.168 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.169 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.170 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.170 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.170 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.170 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.170 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '61abfb08-77b3-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi039') 2024-09-21T00:53:50.170 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config dump 2024-09-21T00:53:50.541 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:50.679 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: Reconfiguring mon.c (monmap changed)... 2024-09-21T00:53:50.679 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: Reconfiguring daemon mon.c on smithi160 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/727596018' entity='client.admin' 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:50.680 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: Reconfiguring mon.c (monmap changed)... 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: Reconfiguring daemon mon.c on smithi160 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/727596018' entity='client.admin' 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: Reconfiguring mon.c (monmap changed)... 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: Reconfiguring daemon mon.c on smithi160 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/727596018' entity='client.admin' 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:53:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.238+0000 7f870fa00640 1 -- 172.21.15.39:0/2400450333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 msgr2=0x7f8708112850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.238+0000 7f870fa00640 1 --2- 172.21.15.39:0/2400450333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f8708112850 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8700009ca0 tx=0x7f870002a8c0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.238+0000 7f870fa00640 1 -- 172.21.15.39:0/2400450333 shutdown_connections 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.238+0000 7f870fa00640 1 --2- 172.21.15.39:0/2400450333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f8708112850 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.238+0000 7f870fa00640 1 --2- 172.21.15.39:0/2400450333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f870810b6d0 0x7f870810bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.239+0000 7f870fa00640 1 --2- 172.21.15.39:0/2400450333 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8708105d90 0x7f87081061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.239+0000 7f870fa00640 1 -- 172.21.15.39:0/2400450333 >> 172.21.15.39:0/2400450333 conn(0x7f8708101860 msgr2=0x7f8708103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:51.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.239+0000 7f870fa00640 1 -- 172.21.15.39:0/2400450333 shutdown_connections 2024-09-21T00:53:51.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.239+0000 7f870fa00640 1 -- 172.21.15.39:0/2400450333 wait complete. 2024-09-21T00:53:51.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 Processor -- start 2024-09-21T00:53:51.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 -- start start 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8708105d90 0x7f870819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f870810b6d0 0x7f870819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f87081a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8708115ce0 con 0x7f870810c020 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8708115b60 con 0x7f870810b6d0 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.240+0000 7f870fa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8708115e60 con 0x7f8708105d90 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f87081a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f87081a1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43328/0 (socket says 172.21.15.39:43328) 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 -- 172.21.15.39:0/1553890356 learned_addr learned my addr 172.21.15.39:0/1553890356 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f8707e00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f870810b6d0 0x7f870819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870d000640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8708105d90 0x7f870819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 -- 172.21.15.39:0/1553890356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8708105d90 msgr2=0x7f870819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8708105d90 0x7f870819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 -- 172.21.15.39:0/1553890356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f870810b6d0 msgr2=0x7f870819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f870810b6d0 0x7f870819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.241+0000 7f870da00640 1 -- 172.21.15.39:0/1553890356 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87081a26e0 con 0x7f870810c020 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.242+0000 7f870da00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f87081a1fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8700005cc0 tx=0x7f8700031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.242+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700003b70 con 0x7f870810c020 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.242+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87000318f0 con 0x7f870810c020 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.242+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87081a2970 con 0x7f870810c020 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.242+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870002f950 con 0x7f870810c020 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.242+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87081a9130 con 0x7f870810c020 2024-09-21T00:53:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.243+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87081061a0 con 0x7f870810c020 2024-09-21T00:53:51.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.245+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f870002fb50 con 0x7f870810c020 2024-09-21T00:53:51.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.247+0000 7f8705600640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f00b6550 0x7f86f00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:51.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.247+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8700106840 con 0x7f870810c020 2024-09-21T00:53:51.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.247+0000 7f870d000640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f00b6550 0x7f86f00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:51.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.248+0000 7f870d000640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f00b6550 0x7f86f00b8a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009bd0 tx=0x7f86fc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:51.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.251+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f87000cec30 con 0x7f870810c020 2024-09-21T00:53:51.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.450+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f870810de70 con 0x7f870810c020 2024-09-21T00:53:51.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.451+0000 7f8705600640 1 -- 172.21.15.39:0/1553890356 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+21721 (secure 0 0 0) 0x7f87000d3010 con 0x7f870810c020 2024-09-21T00:53:51.452 INFO:teuthology.orchestra.run.smithi039.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-21T00:53:51.452 INFO:teuthology.orchestra.run.smithi039.stdout:global dev auth_debug true 2024-09-21T00:53:51.452 INFO:teuthology.orchestra.run.smithi039.stdout:global basic container_image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 * 2024-09-21T00:53:51.452 INFO:teuthology.orchestra.run.smithi039.stdout:global dev debug_asserts_on_shutdown true 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global basic log_to_file true 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global basic log_to_journald false 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global basic log_to_stderr false 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_allow_pool_delete true 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_clock_drift_allowed 1.000000 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_cluster_log_to_file true 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_max_pg_per_osd 10000 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2024-09-21T00:53:51.453 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced mon_warn_on_too_few_osds false 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:global dev ms_die_on_bug true 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:global dev ms_die_on_old_message true 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:global dev ms_die_on_skipped_message false 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced osd_pool_default_erasure_code_profile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced auth_service_ticket_ttl 240.000000 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced debug_mon 20/20 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced debug_ms 1/1 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced debug_paxos 20/20 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_data_avail_warn 5 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_down_mkfs_grace 300 2024-09-21T00:53:51.454 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_mgr_mkfs_grace 240 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_op_complaint_time 120 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon dev mon_osd_prime_pg_temp true 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_osd_reporter_subtree_level osd 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mon advanced public_network 172.21.0.0/20 * 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced cephsqlite_lock_renewal_timeout 900000 2024-09-21T00:53:51.455 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced client_mount_timeout 30 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced debug_cephsqlite 20/20 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced debug_client 20/20 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced debug_mgr 20/20 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced debug_ms 1/1 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/cephadm/container_init True * 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/cephadm/migration_current 7 * 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/snap_schedule/allow_m_granularity true 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/snap_schedule/dump_on_update true 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mgr/snap_schedule/log_level debug 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T00:53:51.456 INFO:teuthology.orchestra.run.smithi039.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev bdev_debug_aio true 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev bluestore_block_size 96636764160 * 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev bluestore_fsck_on_mount true 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced debug_bluefs 20/20 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced debug_bluestore 20/20 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced debug_ms 1/1 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced debug_osd 20/20 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced debug_rocksdb 10/10 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_misdirected_ops true 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_op_order true 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_pg_log_writeout true 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_shutdown true 2024-09-21T00:53:51.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_verify_cached_snaps true 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_verify_missing_on_start true 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd dev osd_debug_verify_stray_on_activate true 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_failsafe_full_ratio 0.950000 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_max_omap_entries_per_request 10000 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_mclock_profile high_recovery_ops 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_memory_target_autotune true 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_objectstore bluestore * 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_op_queue debug_random * 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_op_queue_cut_off debug_random * 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_recover_clone_overlap true 2024-09-21T00:53:51.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_recovery_max_chunk 1048576 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_scrub_load_threshold 5.000000 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_scrub_max_interval 600.000000 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:osd advanced osd_shutdown_pgref_assert true 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced debug_mds 20/20 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced debug_mds_balancer 20/20 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced debug_ms 2/2 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced mds_bal_fragment_size_max 10000 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced mds_bal_merge_size 5 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced mds_bal_split_bits 3 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced mds_bal_split_size 100 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds dev mds_debug_frag true 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds dev mds_debug_scatterstat true 2024-09-21T00:53:51.459 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:mds dev mds_verify_scatter true 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced rados_mon_op_timeout 900 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:mds advanced rados_osd_op_timeout 900 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client advanced client_mount_timeout 600 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client advanced debug_client 20/20 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client advanced debug_ms 1/1 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client advanced rados_mon_op_timeout 900 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client advanced rados_osd_op_timeout 900 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client.rgw advanced rgw_cache_enabled true 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client.rgw advanced rgw_enable_ops_log true 2024-09-21T00:53:51.460 INFO:teuthology.orchestra.run.smithi039.stdout:client.rgw advanced rgw_enable_usage_log true 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f00b6550 msgr2=0x7f86f00b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f00b6550 0x7f86f00b8a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009bd0 tx=0x7f86fc005f90 comp rx=0 tx=0).stop 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 msgr2=0x7f87081a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f87081a1fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8700005cc0 tx=0x7f8700031700 comp rx=0 tx=0).stop 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 shutdown_connections 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f870810c020 0x7f87081a1fb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f00b6550 0x7f86f00b8a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f870810b6d0 0x7f870819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 --2- 172.21.15.39:0/1553890356 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8708105d90 0x7f870819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 >> 172.21.15.39:0/1553890356 conn(0x7f8708101860 msgr2=0x7f8708108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 shutdown_connections 2024-09-21T00:53:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:51.454+0000 7f870fa00640 1 -- 172.21.15.39:0/1553890356 wait complete. 2024-09-21T00:53:51.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:51 smithi039 ceph-mon[30415]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:51 smithi150 ceph-mon[34945]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:51 smithi160 ceph-mon[34696]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:52.485 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T00:53:52.485 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T00:53:52.486 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T00:53:52.521 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T00:53:52.521 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 00:52:15.359086643 +0000 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 00:47:22.797822080 +0000 2024-09-21T00:53:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 00:47:22.797822080 +0000 2024-09-21T00:53:52.598 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-21 00:47:22.797822080 +0000 2024-09-21T00:53:52.598 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T00:53:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T00:53:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T00:53:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000366179 s, 1.4 MB/s 2024-09-21T00:53:52.691 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:52 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1553890356' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T00:53:52.693 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T00:53:52.729 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-09-21T00:53:52.804 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T00:53:52.804 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:52.804 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-09-21T00:53:52.805 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:52.805 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:52.805 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 00:52:15.418085508 +0000 2024-09-21T00:53:52.805 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 00:47:23.284812774 +0000 2024-09-21T00:53:52.805 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 00:47:23.284812774 +0000 2024-09-21T00:53:52.805 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-21 00:47:23.284812774 +0000 2024-09-21T00:53:52.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T00:53:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:52 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1553890356' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T00:53:52.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:52 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1553890356' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T00:53:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T00:53:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T00:53:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000475128 s, 1.1 MB/s 2024-09-21T00:53:52.901 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T00:53:52.937 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-09-21T00:53:53.012 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 00:52:15.459084719 +0000 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 00:47:23.752803831 +0000 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 00:47:23.752803831 +0000 2024-09-21T00:53:53.013 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-21 00:47:23.752803831 +0000 2024-09-21T00:53:53.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T00:53:53.106 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T00:53:53.106 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T00:53:53.106 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000348672 s, 1.5 MB/s 2024-09-21T00:53:53.108 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T00:53:53.144 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-09-21T00:53:53.220 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T00:53:53.220 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:53.220 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-09-21T00:53:53.220 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:53.221 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:53.221 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 00:52:15.498083969 +0000 2024-09-21T00:53:53.221 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 00:47:24.233794641 +0000 2024-09-21T00:53:53.221 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 00:47:24.233794641 +0000 2024-09-21T00:53:53.221 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-21 00:47:24.233794641 +0000 2024-09-21T00:53:53.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T00:53:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T00:53:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T00:53:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000543968 s, 941 kB/s 2024-09-21T00:53:53.317 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T00:53:53.353 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-21T00:53:53.353 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T00:53:53.373 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T00:53:53.373 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-09-21T00:53:53.432 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-21 00:53:18.985724127 +0000 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-21 00:46:56.775332089 +0000 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-21 00:46:56.775332089 +0000 2024-09-21T00:53:53.433 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-21 00:46:56.775332089 +0000 2024-09-21T00:53:53.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T00:53:53.504 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-21T00:53:53.504 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-21T00:53:53.504 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000308137 s, 1.7 MB/s 2024-09-21T00:53:53.505 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T00:53:53.564 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-09-21T00:53:53.621 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-21 00:53:18.985724127 +0000 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-21 00:46:57.099325765 +0000 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-21 00:46:57.099325765 +0000 2024-09-21T00:53:53.622 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-21 00:46:57.099325765 +0000 2024-09-21T00:53:53.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T00:53:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-21T00:53:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-21T00:53:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000301823 s, 1.7 MB/s 2024-09-21T00:53:53.689 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T00:53:53.746 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-09-21T00:53:53.802 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-21 00:53:18.985724127 +0000 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-21 00:46:57.424319422 +0000 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-21 00:46:57.424319422 +0000 2024-09-21T00:53:53.803 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-21 00:46:57.424319422 +0000 2024-09-21T00:53:53.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T00:53:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:53 smithi150 ceph-mon[34945]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:53.838 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-21T00:53:53.838 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-21T00:53:53.838 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000212072 s, 2.4 MB/s 2024-09-21T00:53:53.839 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T00:53:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:53 smithi160 ceph-mon[34696]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:53.896 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-09-21T00:53:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:53 smithi039 ceph-mon[30415]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-21 00:53:18.986724108 +0000 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-21 00:46:57.750313071 +0000 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-21 00:46:57.750313071 +0000 2024-09-21T00:53:53.952 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-09-21 00:46:57.750313071 +0000 2024-09-21T00:53:53.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T00:53:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-21T00:53:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-21T00:53:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000217284 s, 2.4 MB/s 2024-09-21T00:53:54.017 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T00:53:54.074 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T00:53:54.074 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T00:53:54.094 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T00:53:54.094 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 00:53:16.579467849 +0000 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 00:46:43.253561274 +0000 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 00:46:43.253561274 +0000 2024-09-21T00:53:54.153 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 00:46:43.253561274 +0000 2024-09-21T00:53:54.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T00:53:54.224 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T00:53:54.224 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T00:53:54.224 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000289449 s, 1.8 MB/s 2024-09-21T00:53:54.225 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T00:53:54.284 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 00:53:16.579467849 +0000 2024-09-21T00:53:54.341 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 00:46:43.576556480 +0000 2024-09-21T00:53:54.342 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 00:46:43.576556480 +0000 2024-09-21T00:53:54.342 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 00:46:43.576556480 +0000 2024-09-21T00:53:54.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T00:53:54.406 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T00:53:54.407 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T00:53:54.407 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000215934 s, 2.4 MB/s 2024-09-21T00:53:54.408 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T00:53:54.465 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-09-21T00:53:54.521 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T00:53:54.521 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:54.521 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T00:53:54.521 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:54.521 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:54.522 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 00:53:16.580467834 +0000 2024-09-21T00:53:54.522 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 00:46:43.897551716 +0000 2024-09-21T00:53:54.522 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 00:46:43.897551716 +0000 2024-09-21T00:53:54.522 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 00:46:43.897551716 +0000 2024-09-21T00:53:54.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T00:53:54.585 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T00:53:54.585 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T00:53:54.585 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000172537 s, 3.0 MB/s 2024-09-21T00:53:54.587 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T00:53:54.643 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 00:53:16.580467834 +0000 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 00:46:44.232546744 +0000 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 00:46:44.232546744 +0000 2024-09-21T00:53:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 00:46:44.232546744 +0000 2024-09-21T00:53:54.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T00:53:54.763 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T00:53:54.763 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T00:53:54.763 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000205361 s, 2.5 MB/s 2024-09-21T00:53:54.765 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T00:53:54.822 INFO:tasks.cephadm:Deploying osd.0 on smithi039 with /dev/vg_nvme/lv_4... 2024-09-21T00:53:54.822 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T00:53:55.199 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:55.625 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:55 smithi039 ceph-mon[30415]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:55 smithi150 ceph-mon[34945]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:55 smithi160 ceph-mon[34696]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:57 smithi150 ceph-mon[34945]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:57 smithi160 ceph-mon[34696]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:57.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:57 smithi039 ceph-mon[30415]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:58.659 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:53:58.694 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi039:vg_nvme/lv_4 2024-09-21T00:53:59.064 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:53:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 -- 172.21.15.39:0/426221395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818103b70 msgr2=0x7f3818103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:59.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 --2- 172.21.15.39:0/426221395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818103b70 0x7f3818103f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f380c009860 tx=0x7f380c02a340 comp rx=0 tx=0).stop 2024-09-21T00:53:59.733 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:53:59 smithi039 ceph-mon[30415]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 -- 172.21.15.39:0/426221395 shutdown_connections 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 --2- 172.21.15.39:0/426221395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818109e00 0x7f3818110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 --2- 172.21.15.39:0/426221395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38181094b0 0x7f38181098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 --2- 172.21.15.39:0/426221395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818103b70 0x7f3818103f80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 -- 172.21.15.39:0/426221395 >> 172.21.15.39:0/426221395 conn(0x7f38180ff5e0 msgr2=0x7f3818101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 -- 172.21.15.39:0/426221395 shutdown_connections 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.732+0000 7f3820600640 1 -- 172.21.15.39:0/426221395 wait complete. 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.733+0000 7f3820600640 1 Processor -- start 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.733+0000 7f3820600640 1 -- start start 2024-09-21T00:53:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f3820600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 0x7f381819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:59.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f3820600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38181094b0 0x7f381819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:59.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f3820600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818109e00 0x7f381819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:59.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f3820600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3818113ac0 con 0x7f3818103b70 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f3820600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3818113940 con 0x7f38181094b0 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f3820600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3818113c40 con 0x7f3818109e00 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 0x7f381819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 0x7f381819a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52564/0 (socket says 172.21.15.39:52564) 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381dc00640 1 -- 172.21.15.39:0/1540529431 learned_addr learned my addr 172.21.15.39:0/1540529431 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381d200640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38181094b0 0x7f381819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381e600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818109e00 0x7f381819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381dc00640 1 -- 172.21.15.39:0/1540529431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818109e00 msgr2=0x7f381819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.734+0000 7f381dc00640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818109e00 0x7f381819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f381dc00640 1 -- 172.21.15.39:0/1540529431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38181094b0 msgr2=0x7f381819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f381dc00640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38181094b0 0x7f381819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f381dc00640 1 -- 172.21.15.39:0/1540529431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38181a03e0 con 0x7f3818103b70 2024-09-21T00:53:59.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f381dc00640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 0x7f381819a470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f380c0303f0 tx=0x7f380c032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c040020 con 0x7f3818103b70 2024-09-21T00:53:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38181a0670 con 0x7f3818103b70 2024-09-21T00:53:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f380c030dd0 con 0x7f3818103b70 2024-09-21T00:53:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.735+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c037040 con 0x7f3818103b70 2024-09-21T00:53:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.736+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38180755f0 con 0x7f3818103b70 2024-09-21T00:53:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.737+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3818104df0 con 0x7f3818103b70 2024-09-21T00:53:59.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.739+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f380c043020 con 0x7f3818103b70 2024-09-21T00:53:59.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.741+0000 7f3816000640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f38000b6550 0x7f38000b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:53:59.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.741+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f380c105b80 con 0x7f3818103b70 2024-09-21T00:53:59.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.741+0000 7f381d200640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f38000b6550 0x7f38000b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:53:59.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.742+0000 7f381d200640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f38000b6550 0x7f38000b8a60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3818108e60 tx=0x7f3804005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:53:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.748+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f380c04a030 con 0x7f3818103b70 2024-09-21T00:53:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:53:59 smithi150 ceph-mon[34945]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:53:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:53:59 smithi160 ceph-mon[34696]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:00.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:53:59.999+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3818063800 con 0x7f38000b6550 2024-09-21T00:54:00.756 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:00.756 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:00.756 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:01 smithi150 ceph-mon[34945]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:01 smithi150 ceph-mon[34945]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:01 smithi160 ceph-mon[34696]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:01 smithi160 ceph-mon[34696]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:01 smithi039 ceph-mon[30415]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:01.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:01 smithi039 ceph-mon[30415]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:03 smithi150 ceph-mon[34945]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:03 smithi160 ceph-mon[34696]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:03 smithi039 ceph-mon[30415]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:04.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:04 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2504799200' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db1d8552-f65b-468d-b4f6-a019f730b0fd"} : dispatch 2024-09-21T00:54:04.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:04 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2504799200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db1d8552-f65b-468d-b4f6-a019f730b0fd"}]': finished 2024-09-21T00:54:04.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:04 smithi150 ceph-mon[34945]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T00:54:04.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:04 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:04.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:04 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2504799200' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db1d8552-f65b-468d-b4f6-a019f730b0fd"} : dispatch 2024-09-21T00:54:04.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:04 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2504799200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db1d8552-f65b-468d-b4f6-a019f730b0fd"}]': finished 2024-09-21T00:54:04.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:04 smithi160 ceph-mon[34696]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T00:54:04.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:04 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:04 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2504799200' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db1d8552-f65b-468d-b4f6-a019f730b0fd"} : dispatch 2024-09-21T00:54:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:04 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2504799200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db1d8552-f65b-468d-b4f6-a019f730b0fd"}]': finished 2024-09-21T00:54:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:04 smithi039 ceph-mon[30415]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T00:54:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:04 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:05 smithi150 ceph-mon[34945]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:05 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1620688829' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:54:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:05 smithi160 ceph-mon[34696]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:05 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1620688829' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:54:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:05 smithi039 ceph-mon[30415]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:05 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1620688829' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:54:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:07 smithi150 ceph-mon[34945]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:07 smithi160 ceph-mon[34696]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:07 smithi039 ceph-mon[30415]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:09 smithi150 ceph-mon[34945]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:09 smithi160 ceph-mon[34696]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:09 smithi039 ceph-mon[30415]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:11 smithi039 ceph-mon[30415]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:11 smithi150 ceph-mon[34945]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:11 smithi160 ceph-mon[34696]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:13 smithi039 ceph-mon[30415]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:13 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T00:54:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:13 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:13 smithi150 ceph-mon[34945]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:13 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T00:54:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:13 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:13 smithi160 ceph-mon[34696]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:13 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T00:54:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:13 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:14.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:14 smithi150 ceph-mon[34945]: Deploying daemon osd.0 on smithi039 2024-09-21T00:54:14.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:14 smithi039 ceph-mon[30415]: Deploying daemon osd.0 on smithi039 2024-09-21T00:54:14.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:14 smithi160 ceph-mon[34696]: Deploying daemon osd.0 on smithi039 2024-09-21T00:54:15.736 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:15 smithi039 ceph-mon[30415]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:15 smithi150 ceph-mon[34945]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:15.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:15 smithi160 ceph-mon[34696]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:17.579 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:17 smithi039 ceph-mon[30415]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:17 smithi150 ceph-mon[34945]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:17 smithi160 ceph-mon[34696]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:19 smithi150 ceph-mon[34945]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:19 smithi160 ceph-mon[34696]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:19 smithi039 ceph-mon[30415]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:21 smithi150 ceph-mon[34945]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:21 smithi160 ceph-mon[34696]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:21 smithi039 ceph-mon[30415]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:23 smithi150 ceph-mon[34945]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:23 smithi160 ceph-mon[34696]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:23.886 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:23 smithi039 ceph-mon[30415]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:23.886 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:23.886 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:23.887 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:25.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:25 smithi039 ceph-mon[30415]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:25 smithi150 ceph-mon[34945]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:25.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:25 smithi160 ceph-mon[34696]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:27 smithi039 ceph-mon[30415]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:27.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:27.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:27 smithi150 ceph-mon[34945]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:27 smithi160 ceph-mon[34696]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:28.203 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 0 on host 'smithi039' 2024-09-21T00:54:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.201+0000 7f3816000640 1 -- 172.21.15.39:0/1540529431 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3818063800 con 0x7f38000b6550 2024-09-21T00:54:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f38000b6550 msgr2=0x7f38000b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f38000b6550 0x7f38000b8a60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3818108e60 tx=0x7f3804005f90 comp rx=0 tx=0).stop 2024-09-21T00:54:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 msgr2=0x7f381819a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 0x7f381819a470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f380c0303f0 tx=0x7f380c032da0 comp rx=0 tx=0).stop 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 shutdown_connections 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3818109e00 0x7f381819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f38000b6550 0x7f38000b8a60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38181094b0 0x7f381819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 --2- 172.21.15.39:0/1540529431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3818103b70 0x7f381819a470 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 >> 172.21.15.39:0/1540529431 conn(0x7f38180ff5e0 msgr2=0x7f3818101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.204+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 shutdown_connections 2024-09-21T00:54:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:54:28.205+0000 7f3820600640 1 -- 172.21.15.39:0/1540529431 wait complete. 2024-09-21T00:54:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:28.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:28.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:28.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:29.117 DEBUG:teuthology.orchestra.run.smithi039:osd.0> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.0.service 2024-09-21T00:54:29.122 INFO:tasks.cephadm:Deploying osd.1 on smithi150 with /dev/vg_nvme/lv_4... 2024-09-21T00:54:29.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T00:54:29.368 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:54:29.383 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:54:29 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0[46832]: 2024-09-21T00:54:29.204+0000 7f534362e640 -1 osd.0 0 log_to_monitors true 2024-09-21T00:54:29.644 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:29 smithi039 ceph-mon[30415]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:29.645 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:29 smithi039 ceph-mon[30415]: from='osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T00:54:29.645 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:29 smithi039 ceph-mon[30415]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T00:54:29.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:29 smithi150 ceph-mon[34945]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:29.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:29 smithi150 ceph-mon[34945]: from='osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T00:54:29.723 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:29 smithi150 ceph-mon[34945]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T00:54:29.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:29 smithi160 ceph-mon[34696]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:29.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:29 smithi160 ceph-mon[34696]: from='osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T00:54:29.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:29 smithi160 ceph-mon[34696]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: from='osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T00:54:30.626 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:30.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0[46832]: 2024-09-21T00:54:30.445+0000 7f533e200640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T00:54:30.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0[46832]: 2024-09-21T00:54:30.449+0000 7f5339200640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: from='osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T00:54:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:30.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: from='osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T00:54:30.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:31 smithi150 ceph-mon[34945]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:31 smithi039 ceph-mon[30415]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:31.729 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:54:31.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-09-21T00:54:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:31 smithi160 ceph-mon[34696]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:31.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:31.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:32.006 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:54:32.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 -- 172.21.15.150:0/3159001828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 msgr2=0x7f1da4104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 --2- 172.21.15.150:0/3159001828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da4104de0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009ca0 tx=0x7f1d9402a8e0 comp rx=0 tx=0).stop 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 -- 172.21.15.150:0/3159001828 shutdown_connections 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 --2- 172.21.15.150:0/3159001828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da410de30 0x7f1da4110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 --2- 172.21.15.150:0/3159001828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da4105320 0x7f1da410d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 --2- 172.21.15.150:0/3159001828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da4104de0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 -- 172.21.15.150:0/3159001828 >> 172.21.15.150:0/3159001828 conn(0x7f1da4079740 msgr2=0x7f1da4079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 -- 172.21.15.150:0/3159001828 shutdown_connections 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.497+0000 7f1dab73b640 1 -- 172.21.15.150:0/3159001828 wait complete. 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 Processor -- start 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 -- start start 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:32.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da4105320 0x7f1da419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da410de30 0x7f1da419fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1da4113af0 con 0x7f1da4105320 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1da4113970 con 0x7f1da41049d0 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1dab73b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1da4113c70 con 0x7f1da410de30 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1da94d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1da94d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da419a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40792/0 (socket says 172.21.15.150:40792) 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1da94d7640 1 -- 172.21.15.150:0/2133256726 learned_addr learned my addr 172.21.15.150:0/2133256726 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.498+0000 7f1da8cd6640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da4105320 0x7f1da419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da94d7640 1 -- 172.21.15.150:0/2133256726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da410de30 msgr2=0x7f1da419fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da9cd8640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da410de30 0x7f1da419fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da94d7640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da410de30 0x7f1da419fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da94d7640 1 -- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da4105320 msgr2=0x7f1da419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:32.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da94d7640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da4105320 0x7f1da419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da94d7640 1 -- 172.21.15.150:0/2133256726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da41a04b0 con 0x7f1da41049d0 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da8cd6640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da4105320 0x7f1da419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da9cd8640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da410de30 0x7f1da419fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da94d7640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da419a540 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1d9402ae00 tx=0x7f1d94003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94004240 con 0x7f1da41049d0 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da41a0740 con 0x7f1da41049d0 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d94003e50 con 0x7f1da41049d0 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94030eb0 con 0x7f1da41049d0 2024-09-21T00:54:32.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.499+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41a6f00 con 0x7f1da41049d0 2024-09-21T00:54:32.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.500+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da4104de0 con 0x7f1da41049d0 2024-09-21T00:54:32.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.502+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f1d940305e0 con 0x7f1da41049d0 2024-09-21T00:54:32.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.503+0000 7f1da27fc640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1d8c0b64d0 0x7f1d8c0b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:32.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.503+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7f1d94107070 con 0x7f1da41049d0 2024-09-21T00:54:32.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.503+0000 7f1da8cd6640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1d8c0b64d0 0x7f1d8c0b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:32.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.503+0000 7f1da8cd6640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1d8c0b64d0 0x7f1d8c0b89e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1da4079f80 tx=0x7f1d98008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:54:32.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.506+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1d940cf2a0 con 0x7f1da41049d0 2024-09-21T00:54:32.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:32 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:54:32.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:32 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:54:32.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:32 smithi039 ceph-mon[30415]: osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] boot 2024-09-21T00:54:32.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:32 smithi039 ceph-mon[30415]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T00:54:32.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:32.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:32.693+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1da4063800 con 0x7f1d8c0b64d0 2024-09-21T00:54:32.694 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:32 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:54:32.694 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:32 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:54:32.694 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:32 smithi150 ceph-mon[34945]: osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] boot 2024-09-21T00:54:32.694 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:32 smithi150 ceph-mon[34945]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T00:54:32.694 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:32.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:32 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:54:32.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:32 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:54:32.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:32 smithi160 ceph-mon[34696]: osd.0 [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] boot 2024-09-21T00:54:32.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:32 smithi160 ceph-mon[34696]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T00:54:32.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:54:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T00:54:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T00:54:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi039 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 17089M 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:34 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi039 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 17089M 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:34.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:34 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi039 2024-09-21T00:54:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 17089M 2024-09-21T00:54:34.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:34.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:34 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:35 smithi150 ceph-mon[34945]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:35 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/607204785' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"} : dispatch 2024-09-21T00:54:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:35 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"} : dispatch 2024-09-21T00:54:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:35 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"}]': finished 2024-09-21T00:54:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:35 smithi150 ceph-mon[34945]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T00:54:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:35 smithi160 ceph-mon[34696]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:35 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/607204785' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"} : dispatch 2024-09-21T00:54:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:35 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"} : dispatch 2024-09-21T00:54:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:35 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"}]': finished 2024-09-21T00:54:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:35 smithi160 ceph-mon[34696]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T00:54:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:35 smithi039 ceph-mon[30415]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:35 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/607204785' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"} : dispatch 2024-09-21T00:54:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:35 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"} : dispatch 2024-09-21T00:54:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:35 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a09b890-68cb-4443-b9eb-f1bf8557126e"}]': finished 2024-09-21T00:54:35.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:35 smithi039 ceph-mon[30415]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T00:54:35.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:36.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:36 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/528795817' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:54:36.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:36 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/528795817' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:54:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:36 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/528795817' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:54:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:37 smithi150 ceph-mon[34945]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:37.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:37 smithi160 ceph-mon[34696]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:37 smithi039 ceph-mon[30415]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:39 smithi150 ceph-mon[34945]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:39.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:39 smithi160 ceph-mon[34696]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:39 smithi039 ceph-mon[30415]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:41 smithi150 ceph-mon[34945]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:41.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:41 smithi160 ceph-mon[34696]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:41 smithi039 ceph-mon[30415]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:43.758 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:43 smithi150 ceph-mon[34945]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:43.759 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T00:54:43.759 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:43 smithi160 ceph-mon[34696]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T00:54:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:43.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:43 smithi039 ceph-mon[30415]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:43.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T00:54:43.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:44.666 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:44 smithi150 ceph-mon[34945]: Deploying daemon osd.1 on smithi150 2024-09-21T00:54:44.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:44 smithi160 ceph-mon[34696]: Deploying daemon osd.1 on smithi150 2024-09-21T00:54:44.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:44 smithi039 ceph-mon[30415]: Deploying daemon osd.1 on smithi150 2024-09-21T00:54:45.545 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:45 smithi150 ceph-mon[34945]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:45 smithi160 ceph-mon[34696]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:45 smithi039 ceph-mon[30415]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:47 smithi150 ceph-mon[34945]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:47 smithi160 ceph-mon[34696]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:47 smithi039 ceph-mon[30415]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:49 smithi150 ceph-mon[34945]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:49 smithi160 ceph-mon[34696]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:49 smithi039 ceph-mon[30415]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:50.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:51.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:51 smithi150 ceph-mon[34945]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:51.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:51 smithi160 ceph-mon[34696]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:51 smithi039 ceph-mon[30415]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:53 smithi160 ceph-mon[34696]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:53 smithi039 ceph-mon[30415]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:53.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.455 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:53 smithi150 ceph-mon[34945]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:53.455 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.455 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.455 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:53.455 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:53.456 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:53.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.784+0000 7f1da27fc640 1 -- 172.21.15.150:0/2133256726 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1da4063800 con 0x7f1d8c0b64d0 2024-09-21T00:54:53.786 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 1 on host 'smithi150' 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1d8c0b64d0 msgr2=0x7f1d8c0b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1d8c0b64d0 0x7f1d8c0b89e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1da4079f80 tx=0x7f1d98008040 comp rx=0 tx=0).stop 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 msgr2=0x7f1da419a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da419a540 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1d9402ae00 tx=0x7f1d94003ba0 comp rx=0 tx=0).stop 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 shutdown_connections 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1da410de30 0x7f1da419fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da4105320 0x7f1da419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1d8c0b64d0 0x7f1d8c0b89e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 --2- 172.21.15.150:0/2133256726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da41049d0 0x7f1da419a540 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 >> 172.21.15.150:0/2133256726 conn(0x7f1da4079740 msgr2=0x7f1da40ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:54:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 shutdown_connections 2024-09-21T00:54:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:54:53.786+0000 7f1dab73b640 1 -- 172.21.15.150:0/2133256726 wait complete. 2024-09-21T00:54:54.343 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:54.344 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:54.344 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:54.345 DEBUG:teuthology.orchestra.run.smithi150:osd.1> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.1.service 2024-09-21T00:54:54.347 INFO:tasks.cephadm:Deploying osd.2 on smithi160 with /dev/vg_nvme/lv_4... 2024-09-21T00:54:54.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T00:54:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:54:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:54.601 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:54:55.271 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:55 smithi160 ceph-mon[34696]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:55.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:55 smithi039 ceph-mon[30415]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:55.481 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:55 smithi150 ceph-mon[34945]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:55.800 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:54:55 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1[39575]: 2024-09-21T00:54:55.596+0000 7f61fa795640 -1 osd.1 0 log_to_monitors true 2024-09-21T00:54:56.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:56 smithi160 ceph-mon[34696]: from='osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T00:54:56.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:56 smithi160 ceph-mon[34696]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T00:54:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:56 smithi039 ceph-mon[30415]: from='osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T00:54:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:56 smithi039 ceph-mon[30415]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T00:54:56.542 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:56 smithi150 ceph-mon[34945]: from='osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T00:54:56.542 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:56 smithi150 ceph-mon[34945]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T00:54:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:54:56.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_4 2024-09-21T00:54:57.134 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:54:57.206 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1[39575]: 2024-09-21T00:54:57.112+0000 7f61f6717640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T00:54:57.206 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1[39575]: 2024-09-21T00:54:57.116+0000 7f61f2515640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:54:57.206 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-mon[34945]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:57.206 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-mon[34945]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T00:54:57.206 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-mon[34945]: from='osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:54:57.206 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-mon[34945]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T00:54:57.206 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:57.207 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:57 smithi150 ceph-mon[34945]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:54:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:57 smithi039 ceph-mon[30415]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:57 smithi039 ceph-mon[30415]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T00:54:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:57 smithi039 ceph-mon[30415]: from='osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:54:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:57 smithi039 ceph-mon[30415]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T00:54:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:57 smithi039 ceph-mon[30415]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:54:57.451 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:57 smithi160 ceph-mon[34696]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:57.451 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:57 smithi160 ceph-mon[34696]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T00:54:57.451 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:57 smithi160 ceph-mon[34696]: from='osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:54:57.451 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:57 smithi160 ceph-mon[34696]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T00:54:57.451 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:57.451 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:57 smithi160 ceph-mon[34696]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/1289484621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 msgr2=0x7fa8b0103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/1289484621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b0103f70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009900 tx=0x7fa8ac02a410 comp rx=0 tx=0).stop 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/1289484621 shutdown_connections 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/1289484621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 0x7fa8b0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/1289484621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b01094a0 0x7fa8b01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/1289484621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b0103f70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/1289484621 >> 172.21.15.160:0/1289484621 conn(0x7fa8b007b850 msgr2=0x7fa8b007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:54:57.694 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/1289484621 shutdown_connections 2024-09-21T00:54:57.695 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/1289484621 wait complete. 2024-09-21T00:54:57.695 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.692+0000 7fa8b7b3d640 1 Processor -- start 2024-09-21T00:54:57.695 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 -- start start 2024-09-21T00:54:57.696 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:57.696 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b01094a0 0x7fa8b019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 0x7fa8b019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b0113ab0 con 0x7fa8b0109df0 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b0113930 con 0x7fa8b01094a0 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b7b3d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b0113c30 con 0x7fa8b0103b60 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b019a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:55032/0 (socket says 172.21.15.160:55032) 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 -- 172.21.15.160:0/535645436 learned_addr learned my addr 172.21.15.160:0/535645436 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b50d8640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b01094a0 0x7fa8b019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b60da640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 0x7fa8b019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 -- 172.21.15.160:0/535645436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b01094a0 msgr2=0x7fa8b019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b01094a0 0x7fa8b019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 -- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 msgr2=0x7fa8b019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:54:57.697 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 0x7fa8b019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b58d9640 1 -- 172.21.15.160:0/535645436 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b01a0470 con 0x7fa8b0103b60 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.693+0000 7fa8b60da640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 0x7fa8b019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.694+0000 7fa8b58d9640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b019a500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac005940 tx=0x7fa8ac0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.694+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac03f030 con 0x7fa8b0103b60 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.694+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b01a0700 con 0x7fa8b0103b60 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.694+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac003cb0 con 0x7fa8b0103b60 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.694+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac035070 con 0x7fa8b0103b60 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.694+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b01a6ec0 con 0x7fa8b0103b60 2024-09-21T00:54:57.698 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.695+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b0104de0 con 0x7fa8b0103b60 2024-09-21T00:54:57.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.696+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fa8ac042030 con 0x7fa8b0103b60 2024-09-21T00:54:57.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.697+0000 7fa8a6ffd640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa89c0b64d0 0x7fa89c0b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:54:57.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.698+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2244+0+0 (secure 0 0 0) 0x7fa8ac106680 con 0x7fa8b0103b60 2024-09-21T00:54:57.699 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.698+0000 7fa8b50d8640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa89c0b64d0 0x7fa89c0b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:54:57.700 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.698+0000 7fa8b50d8640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa89c0b64d0 0x7fa89c0b89e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8b01064c0 tx=0x7fa8a000b590 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:54:57.702 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.701+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa8ac0ce780 con 0x7fa8b0103b60 2024-09-21T00:54:57.888 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:54:57.887+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa8b0063800 con 0x7fa89c0b64d0 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi150 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi150 to 17088M 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:58.389 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.390 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:58.390 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:58.390 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:58.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:54:58.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T00:54:58.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:58.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi150 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi150 to 17088M 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:58.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi150 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi150 to 17088M 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:54:58.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:54:59.272 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] boot 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T00:54:59.273 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:54:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:54:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:54:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] boot 2024-09-21T00:54:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T00:54:59.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:54:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: osd.1 [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] boot 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T00:54:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:54:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T00:55:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:00 smithi150 ceph-mon[34945]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T00:55:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:00 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/2944471034' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"} : dispatch 2024-09-21T00:55:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:00 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"} : dispatch 2024-09-21T00:55:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:00 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"}]': finished 2024-09-21T00:55:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:00 smithi150 ceph-mon[34945]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T00:55:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:00.567 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:00 smithi160 ceph-mon[34696]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T00:55:00.567 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:00 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/2944471034' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"} : dispatch 2024-09-21T00:55:00.567 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:00 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"} : dispatch 2024-09-21T00:55:00.567 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:00 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"}]': finished 2024-09-21T00:55:00.567 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:00 smithi160 ceph-mon[34696]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T00:55:00.567 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:00 smithi039 ceph-mon[30415]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T00:55:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:00 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/2944471034' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"} : dispatch 2024-09-21T00:55:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:00 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"} : dispatch 2024-09-21T00:55:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:00 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77400671-2855-4f44-911b-98ae802e9162"}]': finished 2024-09-21T00:55:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:00 smithi039 ceph-mon[30415]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T00:55:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:01.554 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:01 smithi039 ceph-mon[30415]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:01.555 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:01 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/1672076287' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:55:01.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:01 smithi150 ceph-mon[34945]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:01.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:01 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/1672076287' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:55:01.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:01 smithi160 ceph-mon[34696]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:01.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:01 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/1672076287' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:55:03.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:03 smithi150 ceph-mon[34945]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:03.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:03 smithi160 ceph-mon[34696]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:03 smithi039 ceph-mon[30415]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:05.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:05 smithi150 ceph-mon[34945]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:05.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:05 smithi160 ceph-mon[34696]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:05 smithi039 ceph-mon[30415]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:07.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:07 smithi150 ceph-mon[34945]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:07.568 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:07 smithi160 ceph-mon[34696]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:07 smithi039 ceph-mon[30415]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:08.376 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T00:55:08.376 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:08.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T00:55:08.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:08.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T00:55:08.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:09.374 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:09 smithi160 ceph-mon[34696]: Deploying daemon osd.2 on smithi160 2024-09-21T00:55:09.374 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:09 smithi160 ceph-mon[34696]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:09.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:09 smithi150 ceph-mon[34945]: Deploying daemon osd.2 on smithi160 2024-09-21T00:55:09.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:09 smithi150 ceph-mon[34945]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:09 smithi039 ceph-mon[30415]: Deploying daemon osd.2 on smithi160 2024-09-21T00:55:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:09 smithi039 ceph-mon[30415]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:11.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:11 smithi150 ceph-mon[34945]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:11 smithi160 ceph-mon[34696]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:11 smithi039 ceph-mon[30415]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:13.560 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:13 smithi160 ceph-mon[34696]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:13 smithi039 ceph-mon[30415]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:13.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:13 smithi150 ceph-mon[34945]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:14.447 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:14.448 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:14.448 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:14.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:14.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:14.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:14.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:14.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:14.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:15.416 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:15 smithi160 ceph-mon[34696]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:15.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:15 smithi039 ceph-mon[30415]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:15 smithi150 ceph-mon[34945]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:17.248 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:16 smithi160 ceph-mon[34696]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:17.248 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:16 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.248 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:16 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.249 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:16 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:17.249 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:16 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:17.249 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:16 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:16 smithi150 ceph-mon[34945]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:16 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:16 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:16 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:16 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:16 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:16 smithi039 ceph-mon[30415]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:17.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:16 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:16 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:16 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:17.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:16 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:17.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:16 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:17.941 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 2 on host 'smithi160' 2024-09-21T00:55:17.942 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.939+0000 7fa8a6ffd640 1 -- 172.21.15.160:0/535645436 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa8b0063800 con 0x7fa89c0b64d0 2024-09-21T00:55:17.944 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa89c0b64d0 msgr2=0x7fa89c0b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:17.944 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa89c0b64d0 0x7fa89c0b89e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8b01064c0 tx=0x7fa8a000b590 comp rx=0 tx=0).stop 2024-09-21T00:55:17.944 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 msgr2=0x7fa8b019a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:17.944 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b019a500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac005940 tx=0x7fa8ac0042f0 comp rx=0 tx=0).stop 2024-09-21T00:55:17.944 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 shutdown_connections 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8b0109df0 0x7fa8b019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa89c0b64d0 0x7fa89c0b89e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b01094a0 0x7fa8b019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 --2- 172.21.15.160:0/535645436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa8b0103b60 0x7fa8b019a500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 >> 172.21.15.160:0/535645436 conn(0x7fa8b007b850 msgr2=0x7fa8b01075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 shutdown_connections 2024-09-21T00:55:17.945 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:55:17.942+0000 7fa8b7b3d640 1 -- 172.21.15.160:0/535645436 wait complete. 2024-09-21T00:55:18.263 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:18.264 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:18.264 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:18.569 DEBUG:teuthology.orchestra.run.smithi160:osd.2> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.2.service 2024-09-21T00:55:18.571 INFO:tasks.cephadm:Deploying osd.3 on smithi039 with /dev/vg_nvme/lv_3... 2024-09-21T00:55:18.572 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T00:55:18.957 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:55:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:18 smithi150 ceph-mon[34945]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:19.348 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:18 smithi039 ceph-mon[30415]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:19.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:18 smithi160 ceph-mon[34696]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:19 smithi160 ceph-mon[34696]: from='osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T00:55:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:19 smithi160 ceph-mon[34696]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T00:55:20.107 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2[39902]: 2024-09-21T00:55:19.888+0000 7fb9ae98b640 -1 osd.2 0 log_to_monitors true 2024-09-21T00:55:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:19 smithi039 ceph-mon[30415]: from='osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T00:55:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:19 smithi039 ceph-mon[30415]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T00:55:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:19 smithi150 ceph-mon[34945]: from='osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T00:55:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:19 smithi150 ceph-mon[34945]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T00:55:21.107 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2[39902]: 2024-09-21T00:55:20.976+0000 7fb9aa90d640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T00:55:21.107 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2[39902]: 2024-09-21T00:55:20.984+0000 7fb9a670b640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:55:21.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-mon[34696]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:21.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-mon[34696]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T00:55:21.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-mon[34696]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T00:55:21.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:21.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-mon[34696]: from='osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:55:21.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:20 smithi160 ceph-mon[34696]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:55:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:20 smithi039 ceph-mon[30415]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:20 smithi039 ceph-mon[30415]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T00:55:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:20 smithi039 ceph-mon[30415]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T00:55:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:20 smithi039 ceph-mon[30415]: from='osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:55:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:20 smithi039 ceph-mon[30415]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:55:21.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:20 smithi150 ceph-mon[34945]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:21.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:20 smithi150 ceph-mon[34945]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T00:55:21.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:20 smithi150 ceph-mon[34945]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T00:55:21.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:21.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:20 smithi150 ceph-mon[34945]: from='osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:55:21.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:20 smithi150 ceph-mon[34945]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:55:22.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:55:22.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T00:55:22.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:22.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:22.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi160 2024-09-21T00:55:22.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:22.132 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi160 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:55:22.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-21T00:55:22.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:22.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:22.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi160 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-21T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:22.962 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:55:22.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi039:vg_nvme/lv_3 2024-09-21T00:55:23.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:55:23.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:55:23.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:23.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:23.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] boot 2024-09-21T00:55:23.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T00:55:23.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:23.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:22 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T00:55:23.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:55:23.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:55:23.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:23.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:23.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] boot 2024-09-21T00:55:23.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T00:55:23.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:23.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:22 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: osd.2 [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] boot 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T00:55:23.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:22 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T00:55:23.369 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:55:24.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 -- 172.21.15.39:0/2806125037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810b4f0 msgr2=0x7f5ed810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 --2- 172.21.15.39:0/2806125037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810b4f0 0x7f5ed810b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009900 tx=0x7f5ec402a3f0 comp rx=0 tx=0).stop 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 -- 172.21.15.39:0/2806125037 shutdown_connections 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 --2- 172.21.15.39:0/2806125037 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ed810be40 0x7f5ed8112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 --2- 172.21.15.39:0/2806125037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810b4f0 0x7f5ed810b900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 --2- 172.21.15.39:0/2806125037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed8105bb0 0x7f5ed8105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 -- 172.21.15.39:0/2806125037 >> 172.21.15.39:0/2806125037 conn(0x7f5ed8101680 msgr2=0x7f5ed8103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 -- 172.21.15.39:0/2806125037 shutdown_connections 2024-09-21T00:55:24.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.098+0000 7f5edf000640 1 -- 172.21.15.39:0/2806125037 wait complete. 2024-09-21T00:55:24.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.099+0000 7f5edf000640 1 Processor -- start 2024-09-21T00:55:24.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.099+0000 7f5edf000640 1 -- start start 2024-09-21T00:55:24.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edf000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ed8105bb0 0x7f5ed819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edf000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 0x7f5ed819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edf000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810be40 0x7f5ed81a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edf000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed8115b00 con 0x7f5ed810b4f0 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edf000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed8115980 con 0x7f5ed810be40 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edf000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed8115c80 con 0x7f5ed8105bb0 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 0x7f5ed819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 0x7f5ed819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60832/0 (socket says 172.21.15.39:60832) 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5edd200640 1 -- 172.21.15.39:0/2937445613 learned_addr learned my addr 172.21.15.39:0/2937445613 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5ede600640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810be40 0x7f5ed81a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:24.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.100+0000 7f5eddc00640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ed8105bb0 0x7f5ed819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:24.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edd200640 1 -- 172.21.15.39:0/2937445613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ed8105bb0 msgr2=0x7f5ed819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:24.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edd200640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ed8105bb0 0x7f5ed819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:24.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edd200640 1 -- 172.21.15.39:0/2937445613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810be40 msgr2=0x7f5ed81a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:24.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edd200640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810be40 0x7f5ed81a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:24.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edd200640 1 -- 172.21.15.39:0/2937445613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed81a2460 con 0x7f5ed810b4f0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5ede600640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810be40 0x7f5ed81a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edd200640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 0x7f5ed819ca90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5ec402a910 tx=0x7f5ec4031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec403f030 con 0x7f5ed810b4f0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed81a26f0 con 0x7f5ed810b4f0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ec4004490 con 0x7f5ed810b4f0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4035070 con 0x7f5ed810b4f0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.101+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed81a8eb0 con 0x7f5ed810b4f0 2024-09-21T00:55:24.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.102+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed8106e30 con 0x7f5ed810b4f0 2024-09-21T00:55:24.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.105+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f5ec4042030 con 0x7f5ed810b4f0 2024-09-21T00:55:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.106+0000 7f5ed6000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5ec00b64d0 0x7f5ec00b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.106+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3441+0+0 (secure 0 0 0) 0x7f5ec41066c0 con 0x7f5ed810b4f0 2024-09-21T00:55:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.107+0000 7f5eddc00640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5ec00b64d0 0x7f5ec00b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.107+0000 7f5eddc00640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5ec00b64d0 0x7f5ec00b89e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0095f0 tx=0x7f5ecc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:55:24.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.109+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5ec40d2670 con 0x7f5ed810b4f0 2024-09-21T00:55:24.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:24.327+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5ed8063800 con 0x7f5ec00b64d0 2024-09-21T00:55:24.473 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T00:55:24.474 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 ceph-mon[34696]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T00:55:24.474 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T00:55:24.559 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T00:55:24.560 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 ceph-mon[30415]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T00:55:24.560 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51108]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51108]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51108]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51108]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51112]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51112]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51112]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51112]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T00:55:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 ceph-mon[34945]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T00:55:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T00:55:24.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:55:24.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43529]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43529]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43532]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:55:24.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43532]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.565 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43532]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.565 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43532]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.857 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43563]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:55:24.857 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43563]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.857 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43563]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.857 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43563]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.858 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43566]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:55:24.858 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.858 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43566]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.858 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43566]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43569]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43569]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43569]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43569]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43572]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43572]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43572]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.859 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:24 smithi160 sudo[43572]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51149]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51149]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51149]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51152]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51152]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51152]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:24.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:24 smithi039 sudo[51152]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43535]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43535]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43535]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43538]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:55:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:24 smithi150 sudo[43538]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='client.14481 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:25 smithi039 ceph-mon[30415]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T00:55:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T00:55:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='client.14481 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:55:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:25 smithi150 ceph-mon[34945]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='client.14481 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:55:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:55:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:55:25.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:55:25.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:55:25.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:25 smithi160 ceph-mon[34696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:55:26.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:26.234+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f5ec40c9eb0 con 0x7f5ed810b4f0 2024-09-21T00:55:26.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:26 smithi150 ceph-mon[34945]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T00:55:26.564 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:26 smithi039 ceph-mon[30415]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T00:55:26.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:26 smithi160 ceph-mon[34696]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T00:55:27.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:27 smithi039 ceph-mon[30415]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:27.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:27 smithi039 ceph-mon[30415]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-09-21T00:55:27.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:27 smithi150 ceph-mon[34945]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:27.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:27 smithi150 ceph-mon[34945]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-09-21T00:55:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:27 smithi160 ceph-mon[34696]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:27 smithi160 ceph-mon[34696]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-09-21T00:55:28.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:28 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2932092942' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f906a00-043e-4282-8c8d-47b889db0c65"} : dispatch 2024-09-21T00:55:28.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:28 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2932092942' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f906a00-043e-4282-8c8d-47b889db0c65"}]': finished 2024-09-21T00:55:28.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:28 smithi150 ceph-mon[34945]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T00:55:28.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:28.571 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:28 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2932092942' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f906a00-043e-4282-8c8d-47b889db0c65"} : dispatch 2024-09-21T00:55:28.571 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:28 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2932092942' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f906a00-043e-4282-8c8d-47b889db0c65"}]': finished 2024-09-21T00:55:28.572 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:28 smithi160 ceph-mon[34696]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T00:55:28.572 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:28.575 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:28 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2932092942' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f906a00-043e-4282-8c8d-47b889db0c65"} : dispatch 2024-09-21T00:55:28.575 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:28 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2932092942' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f906a00-043e-4282-8c8d-47b889db0c65"}]': finished 2024-09-21T00:55:28.575 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:28 smithi039 ceph-mon[30415]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T00:55:28.575 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:29.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:29 smithi150 ceph-mon[34945]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:29.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:29 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4049568401' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:55:29.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:29 smithi160 ceph-mon[34696]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:29.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:29 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4049568401' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:55:29.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:29 smithi039 ceph-mon[30415]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:29.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:29 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4049568401' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:55:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:31 smithi150 ceph-mon[34945]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:31.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:31 smithi160 ceph-mon[34696]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:31 smithi039 ceph-mon[30415]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:33.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:33 smithi039 ceph-mon[30415]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:33 smithi150 ceph-mon[34945]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:33.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:33 smithi160 ceph-mon[34696]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:35.324 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:35 smithi039 ceph-mon[30415]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:35 smithi150 ceph-mon[34945]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:35.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:35 smithi160 ceph-mon[34696]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:37.518 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:37 smithi039 ceph-mon[30415]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:37.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:37 smithi150 ceph-mon[34945]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:37.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:37 smithi160 ceph-mon[34696]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:39 smithi150 ceph-mon[34945]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T00:55:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:39 smithi150 ceph-mon[34945]: Deploying daemon osd.3 on smithi039 2024-09-21T00:55:39.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:39 smithi160 ceph-mon[34696]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T00:55:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:39 smithi160 ceph-mon[34696]: Deploying daemon osd.3 on smithi039 2024-09-21T00:55:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:39 smithi039 ceph-mon[30415]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T00:55:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:39 smithi039 ceph-mon[30415]: Deploying daemon osd.3 on smithi039 2024-09-21T00:55:41.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:41 smithi039 ceph-mon[30415]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:41.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:41 smithi150 ceph-mon[34945]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:41.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:41 smithi160 ceph-mon[34696]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:43.510 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:43 smithi039 ceph-mon[30415]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:43.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:43 smithi150 ceph-mon[34945]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:43.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:43 smithi160 ceph-mon[34696]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:45.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:45 smithi150 ceph-mon[34945]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:45.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:45 smithi160 ceph-mon[34696]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:45.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:45 smithi039 ceph-mon[30415]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:47.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:47 smithi150 ceph-mon[34945]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:47.570 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:47 smithi039 ceph-mon[30415]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:47.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:47 smithi160 ceph-mon[34696]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:48.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:48.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:48.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:48.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:48.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:48.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:48.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:48.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:48.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:49 smithi039 ceph-mon[30415]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:49.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:49 smithi150 ceph-mon[34945]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:49.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:49 smithi160 ceph-mon[34696]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:51.494 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:51 smithi039 ceph-mon[30415]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:51.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:51 smithi150 ceph-mon[34945]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:51.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:51 smithi160 ceph-mon[34696]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:53 smithi150 ceph-mon[34945]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:53 smithi160 ceph-mon[34696]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:53 smithi039 ceph-mon[30415]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:53.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:53.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:55:53.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:55:53.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:54.861 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 3 on host 'smithi039' 2024-09-21T00:55:54.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.859+0000 7f5ed6000640 1 -- 172.21.15.39:0/2937445613 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5ed8063800 con 0x7f5ec00b64d0 2024-09-21T00:55:54.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5ec00b64d0 msgr2=0x7f5ec00b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:54.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5ec00b64d0 0x7f5ec00b89e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0095f0 tx=0x7f5ecc005f90 comp rx=0 tx=0).stop 2024-09-21T00:55:54.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 msgr2=0x7f5ed819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:54.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 0x7f5ed819ca90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5ec402a910 tx=0x7f5ec4031a20 comp rx=0 tx=0).stop 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 shutdown_connections 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5ec00b64d0 0x7f5ec00b89e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ed810be40 0x7f5ed81a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed810b4f0 0x7f5ed819ca90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 --2- 172.21.15.39:0/2937445613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ed8105bb0 0x7f5ed819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 >> 172.21.15.39:0/2937445613 conn(0x7f5ed8101680 msgr2=0x7f5ed8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 shutdown_connections 2024-09-21T00:55:54.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:55:54.862+0000 7f5edf000640 1 -- 172.21.15.39:0/2937445613 wait complete. 2024-09-21T00:55:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:55 smithi150 ceph-mon[34945]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:55 smithi150 ceph-mon[34945]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T00:55:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:55 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:55 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:55 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:55.578 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:55 smithi039 ceph-mon[30415]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:55.579 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:55 smithi039 ceph-mon[30415]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T00:55:55.579 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:55 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:55.579 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:55 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:55.579 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:55 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:55.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:55 smithi160 ceph-mon[34696]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:55.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:55 smithi160 ceph-mon[34696]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T00:55:55.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:55 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:55:55.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:55 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:55.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:55 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:55:55.614 DEBUG:teuthology.orchestra.run.smithi039:osd.3> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.3.service 2024-09-21T00:55:55.618 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_3... 2024-09-21T00:55:55.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T00:55:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:55:56.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:56 smithi150 ceph-mon[34945]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T00:55:56.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:56 smithi150 ceph-mon[34945]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T00:55:56.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:56 smithi150 ceph-mon[34945]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:55:56.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:56.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:56 smithi160 ceph-mon[34696]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T00:55:56.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:56 smithi160 ceph-mon[34696]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T00:55:56.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:56 smithi160 ceph-mon[34696]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:55:56.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:56 smithi039 ceph-mon[30415]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T00:55:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:56 smithi039 ceph-mon[30415]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T00:55:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:56 smithi039 ceph-mon[30415]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:55:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.153 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3[53265]: 2024-09-21T00:55:57.024+0000 7fadbbc00640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T00:55:57.154 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3[53265]: 2024-09-21T00:55:57.045+0000 7fadb1c00640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:55:57.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:57.289 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:55:57.290 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T00:55:57.290 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.290 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.290 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' 2024-09-21T00:55:57.290 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] boot 2024-09-21T00:55:57.290 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:57 smithi150 ceph-mon[34945]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] boot 2024-09-21T00:55:57.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:57 smithi039 ceph-mon[30415]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T00:55:57.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: from='osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449]' entity='osd.3' 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: osd.3 [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] boot 2024-09-21T00:55:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:57 smithi160 ceph-mon[34696]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T00:55:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:58 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:55:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:58 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:55:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:58.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:58 smithi150 ceph-mon[34945]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T00:55:58.579 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:55:58.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:58 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:55:58.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:58 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:55:58.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:58.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:58 smithi160 ceph-mon[34696]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T00:55:58.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-09-21T00:55:58.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:58 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:55:58.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:58 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:55:58.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T00:55:58.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:58 smithi039 ceph-mon[30415]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T00:55:58.860 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:55:59.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 -- 172.21.15.150:0/1504909986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8101f70 msgr2=0x7f1cb8102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:59.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 --2- 172.21.15.150:0/1504909986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8009900 tx=0x7f1ca802a410 comp rx=0 tx=0).stop 2024-09-21T00:55:59.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 -- 172.21.15.150:0/1504909986 shutdown_connections 2024-09-21T00:55:59.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 --2- 172.21.15.150:0/1504909986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cb8109e00 0x7f1cb8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:59.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 --2- 172.21.15.150:0/1504909986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb81094b0 0x7f1cb81098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:59.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 --2- 172.21.15.150:0/1504909986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 -- 172.21.15.150:0/1504909986 >> 172.21.15.150:0/1504909986 conn(0x7f1cb8077c60 msgr2=0x7f1cb80780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 -- 172.21.15.150:0/1504909986 shutdown_connections 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 -- 172.21.15.150:0/1504909986 wait complete. 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.362+0000 7f1cbd411640 1 Processor -- start 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 -- start start 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cb81094b0 0x7f1cb8103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 0x7f1cb8103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8113ac0 con 0x7f1cb8101f70 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8113940 con 0x7f1cb8109e00 2024-09-21T00:55:59.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cbd411640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8113c40 con 0x7f1cb81094b0 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 0x7f1cb8103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 0x7f1cb8103990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34270/0 (socket says 172.21.15.150:34270) 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 -- 172.21.15.150:0/194234797 learned_addr learned my addr 172.21.15.150:0/194234797 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb67fc640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cb81094b0 0x7f1cb8103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb6ffd640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 -- 172.21.15.150:0/194234797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cb81094b0 msgr2=0x7f1cb8103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cb81094b0 0x7f1cb8103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 -- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb8101f70 msgr2=0x7f1cb8102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb77fe640 1 -- 172.21.15.150:0/194234797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb81a8730 con 0x7f1cb8109e00 2024-09-21T00:55:59.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.363+0000 7f1cb6ffd640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.364+0000 7f1cb77fe640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 0x7f1cb8103990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000eaa0 tx=0x7f1cb000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.364+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000cdb0 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.364+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb81a8a20 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.364+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb0005020 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.364+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00116d0 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.364+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb81a8f60 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.365+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb8102380 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.367+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f1cb00051d0 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.368+0000 7f1c9ffff640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca00b6560 0x7f1ca00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.368+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3884+0+0 (secure 0 0 0) 0x7f1cb00e6390 con 0x7f1cb8109e00 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.369+0000 7f1cb6ffd640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca00b6560 0x7f1ca00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:55:59.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.369+0000 7f1cb6ffd640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca00b6560 0x7f1ca00b8a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000c00 tx=0x7f1ca8009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:55:59.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.373+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1cb0016030 con 0x7f1cb8109e00 2024-09-21T00:55:59.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:55:59.562+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1cb8063800 con 0x7f1ca00b6560 2024-09-21T00:55:59.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:55:59 smithi150 ceph-mon[34945]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:55:59.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:55:59 smithi160 ceph-mon[34696]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:55:59.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:55:59 smithi039 ceph-mon[30415]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='client.24434 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:00.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:00.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='client.24434 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:00.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:00.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T00:56:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='client.24434 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:00.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:01.506 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:01 smithi150 ceph-mon[34945]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:01.506 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:01 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi039 2024-09-21T00:56:01.506 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:01 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 8544M 2024-09-21T00:56:01.555 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:01 smithi039 ceph-mon[30415]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:01.555 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:01 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi039 2024-09-21T00:56:01.555 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:01 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 8544M 2024-09-21T00:56:01.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:01 smithi160 ceph-mon[34696]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:01.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:01 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi039 2024-09-21T00:56:01.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:01 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 8544M 2024-09-21T00:56:02.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:02 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/2522612771' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"} : dispatch 2024-09-21T00:56:02.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:02 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"} : dispatch 2024-09-21T00:56:02.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:02 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"}]': finished 2024-09-21T00:56:02.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:02 smithi150 ceph-mon[34945]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T00:56:02.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:02.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:02 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/2522612771' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"} : dispatch 2024-09-21T00:56:02.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:02 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"} : dispatch 2024-09-21T00:56:02.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:02 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"}]': finished 2024-09-21T00:56:02.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:02 smithi160 ceph-mon[34696]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T00:56:02.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:02.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:02 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/2522612771' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"} : dispatch 2024-09-21T00:56:02.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:02 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"} : dispatch 2024-09-21T00:56:02.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:02 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a50d102-41d4-41ec-a5b8-1d40420d3cb7"}]': finished 2024-09-21T00:56:02.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:02 smithi039 ceph-mon[30415]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T00:56:02.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:03.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:03 smithi150 ceph-mon[34945]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:03.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:03 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/202299158' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:03.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:03 smithi160 ceph-mon[34696]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:03.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:03 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/202299158' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:03 smithi039 ceph-mon[30415]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:03 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/202299158' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:05.609 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:05 smithi160 ceph-mon[34696]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:05 smithi039 ceph-mon[30415]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:05 smithi150 ceph-mon[34945]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:07.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:07 smithi150 ceph-mon[34945]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-21T00:56:07.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:07 smithi160 ceph-mon[34696]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-21T00:56:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:07 smithi039 ceph-mon[30415]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-21T00:56:09.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:09 smithi150 ceph-mon[34945]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T00:56:09.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:09 smithi160 ceph-mon[34696]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T00:56:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:09 smithi039 ceph-mon[30415]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T00:56:10.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T00:56:10.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:10.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T00:56:10.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:10.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T00:56:10.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:11.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:11 smithi160 ceph-mon[34696]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:56:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:11 smithi160 ceph-mon[34696]: Deploying daemon osd.4 on smithi150 2024-09-21T00:56:11.626 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:11 smithi039 ceph-mon[30415]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:56:11.626 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:11 smithi039 ceph-mon[30415]: Deploying daemon osd.4 on smithi150 2024-09-21T00:56:11.714 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:11 smithi150 ceph-mon[34945]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:56:11.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:11 smithi150 ceph-mon[34945]: Deploying daemon osd.4 on smithi150 2024-09-21T00:56:13.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:13 smithi160 ceph-mon[34696]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:56:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:13 smithi039 ceph-mon[30415]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:56:13.692 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:13 smithi150 ceph-mon[34945]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:56:15.474 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:15 smithi150 ceph-mon[34945]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T00:56:15.614 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:15 smithi160 ceph-mon[34696]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T00:56:15.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:15 smithi039 ceph-mon[30415]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T00:56:17.595 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:17 smithi150 ceph-mon[34945]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:56:17.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:17.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:17.596 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:17.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:17 smithi160 ceph-mon[34696]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:56:17.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:17.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:17.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:17 smithi039 ceph-mon[30415]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:56:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:19.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:19 smithi160 ceph-mon[34696]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:19.639 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:19 smithi150 ceph-mon[34945]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:19.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:19 smithi039 ceph-mon[30415]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:20.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:20.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:20.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.087+0000 7f1c9ffff640 1 -- 172.21.15.150:0/194234797 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1cb8063800 con 0x7f1ca00b6560 2024-09-21T00:56:21.088 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca00b6560 msgr2=0x7f1ca00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca00b6560 0x7f1ca00b8a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000c00 tx=0x7f1ca8009680 comp rx=0 tx=0).stop 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 msgr2=0x7f1cb8103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 0x7f1cb8103990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000eaa0 tx=0x7f1cb000ef60 comp rx=0 tx=0).stop 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 shutdown_connections 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca00b6560 0x7f1ca00b8a70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cb8109e00 0x7f1cb8103990 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cb81094b0 0x7f1cb8103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 --2- 172.21.15.150:0/194234797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cb8101f70 0x7f1cb8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 >> 172.21.15.150:0/194234797 conn(0x7f1cb8077c60 msgr2=0x7f1cb8107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:56:21.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 shutdown_connections 2024-09-21T00:56:21.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:56:21.089+0000 7f1cbd411640 1 -- 172.21.15.150:0/194234797 wait complete. 2024-09-21T00:56:21.697 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.4.service 2024-09-21T00:56:21.700 INFO:tasks.cephadm:Deploying osd.5 on smithi160 with /dev/vg_nvme/lv_3... 2024-09-21T00:56:21.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T00:56:21.721 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:21 smithi150 ceph-mon[34945]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:21.721 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:21.721 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.721 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:21 smithi160 ceph-mon[34696]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:21 smithi039 ceph-mon[30415]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:21.972 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:56:22.814 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 00:56:22 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T00:56:22.560+0000 7f0fe376c640 -1 osd.4 0 log_to_monitors true 2024-09-21T00:56:23.636 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:23 smithi160 ceph-mon[34696]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:23.636 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:23 smithi160 ceph-mon[34696]: from='osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T00:56:23.636 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:23 smithi160 ceph-mon[34696]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T00:56:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:23 smithi150 ceph-mon[34945]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:23 smithi150 ceph-mon[34945]: from='osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T00:56:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:23 smithi150 ceph-mon[34945]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T00:56:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:23 smithi039 ceph-mon[30415]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:23 smithi039 ceph-mon[30415]: from='osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T00:56:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:23 smithi039 ceph-mon[30415]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T00:56:24.313 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:56:24.338 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_3 2024-09-21T00:56:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:24 smithi150 ceph-mon[34945]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T00:56:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:24 smithi150 ceph-mon[34945]: from='osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:56:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:24 smithi150 ceph-mon[34945]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T00:56:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:24.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:24 smithi150 ceph-mon[34945]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:56:24.596 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:56:24.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:24 smithi160 ceph-mon[34696]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T00:56:24.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:24 smithi160 ceph-mon[34696]: from='osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:56:24.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:24 smithi160 ceph-mon[34696]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T00:56:24.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:24.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:24 smithi160 ceph-mon[34696]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:56:24.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:24 smithi039 ceph-mon[30415]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T00:56:24.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:24 smithi039 ceph-mon[30415]: from='osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:56:24.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:24 smithi039 ceph-mon[30415]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T00:56:24.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:24.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:24 smithi039 ceph-mon[30415]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:56:25.105 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.101+0000 7f1570a55640 1 -- 172.21.15.160:0/4071098519 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c103140 msgr2=0x7f156c103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:25.105 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.101+0000 7f1570a55640 1 --2- 172.21.15.160:0/4071098519 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c103140 0x7f156c103550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1564009900 tx=0x7f156402a410 comp rx=0 tx=0).stop 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.101+0000 7f1570a55640 1 -- 172.21.15.160:0/4071098519 shutdown_connections 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 --2- 172.21.15.160:0/4071098519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c10f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 --2- 172.21.15.160:0/4071098519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f156c1083b0 0x7f156c1087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 --2- 172.21.15.160:0/4071098519 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c103140 0x7f156c103550 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 -- 172.21.15.160:0/4071098519 >> 172.21.15.160:0/4071098519 conn(0x7f156c077c60 msgr2=0x7f156c0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 -- 172.21.15.160:0/4071098519 shutdown_connections 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 -- 172.21.15.160:0/4071098519 wait complete. 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 Processor -- start 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.102+0000 7f1570a55640 1 -- start start 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1570a55640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f156c103140 0x7f156c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1570a55640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c1083b0 0x7f156c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1570a55640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c19fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1570a55640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f156c112a70 con 0x7f156c103140 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1570a55640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f156c1128f0 con 0x7f156c108d00 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1570a55640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f156c112bf0 con 0x7f156c1083b0 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1569d9b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c1083b0 0x7f156c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c19fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c19fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.160:53828/0 (socket says 172.21.15.160:53828) 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 -- 172.21.15.160:0/1868099535 learned_addr learned my addr 172.21.15.160:0/1868099535 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f1569d9b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c1083b0 0x7f156c19aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:33874/0 (socket says 172.21.15.160:33874) 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 -- 172.21.15.160:0/1868099535 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c1083b0 msgr2=0x7f156c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c1083b0 0x7f156c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 -- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f156c103140 msgr2=0x7f156c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:25.107 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156a59c640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f156c103140 0x7f156c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f156c103140 0x7f156c19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.103+0000 7f156ad9d640 1 -- 172.21.15.160:0/1868099535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c1a04c0 con 0x7f156c108d00 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.104+0000 7f156ad9d640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c19fd90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f155c00e970 tx=0x7f155c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.104+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c00ccc0 con 0x7f156c108d00 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.104+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155c00ce60 con 0x7f156c108d00 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.104+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c010430 con 0x7f156c108d00 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.105+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156c1a07b0 con 0x7f156c108d00 2024-09-21T00:56:25.108 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.105+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156c1a6f10 con 0x7f156c108d00 2024-09-21T00:56:25.109 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.106+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156c103550 con 0x7f156c108d00 2024-09-21T00:56:25.109 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.108+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f155c004fb0 con 0x7f156c108d00 2024-09-21T00:56:25.109 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.108+0000 7f155b7fe640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15540b66b0 0x7f15540b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:25.109 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.108+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4036+0+0 (secure 0 0 0) 0x7f155c0e6860 con 0x7f156c108d00 2024-09-21T00:56:25.109 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.109+0000 7f156a59c640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15540b66b0 0x7f15540b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:25.110 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.109+0000 7f156a59c640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15540b66b0 0x7f15540b8bc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f156402a930 tx=0x7f1564005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:56:25.113 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.112+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f155c056710 con 0x7f156c108d00 2024-09-21T00:56:25.307 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:25.306+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f156c063800 con 0x7f15540b66b0 2024-09-21T00:56:25.564 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T00:56:25.280+0000 7f0fdf6ee640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T00:56:25.564 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T00:56:25.285+0000 7f0fd74e4640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:56:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:56:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T00:56:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='osd.4 ' entity='osd.4' 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:25.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='osd.4 ' entity='osd.4' 2024-09-21T00:56:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:25.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='osd.4 ' entity='osd.4' 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:25.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi150 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi150 to 8544M 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: from='client.24461 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] boot 2024-09-21T00:56:26.780 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T00:56:26.781 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi150 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi150 to 8544M 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: from='client.24461 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] boot 2024-09-21T00:56:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T00:56:26.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:56:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:56:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi150 2024-09-21T00:56:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi150 to 8544M 2024-09-21T00:56:26.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: from='client.24461 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:26.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:26.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: osd.4 [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] boot 2024-09-21T00:56:26.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T00:56:26.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T00:56:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:27 smithi160 ceph-mon[34696]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:27 smithi160 ceph-mon[34696]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T00:56:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:27 smithi150 ceph-mon[34945]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:27 smithi150 ceph-mon[34945]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T00:56:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:27 smithi039 ceph-mon[30415]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T00:56:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:27 smithi039 ceph-mon[30415]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T00:56:28.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:28 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/131544681' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"} : dispatch 2024-09-21T00:56:28.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:28 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"} : dispatch 2024-09-21T00:56:28.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:28 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"}]': finished 2024-09-21T00:56:28.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:28 smithi160 ceph-mon[34696]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T00:56:28.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:28.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:28 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/4014288426' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:28 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/131544681' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"} : dispatch 2024-09-21T00:56:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:28 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"} : dispatch 2024-09-21T00:56:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:28 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"}]': finished 2024-09-21T00:56:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:28 smithi150 ceph-mon[34945]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T00:56:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:28 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/4014288426' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:28.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:28 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/131544681' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"} : dispatch 2024-09-21T00:56:28.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:28 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"} : dispatch 2024-09-21T00:56:28.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:28 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64041b91-4bdc-4e60-b280-2407dc0285f0"}]': finished 2024-09-21T00:56:28.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:28 smithi039 ceph-mon[30415]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T00:56:28.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:28.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:28 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/4014288426' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:29 smithi150 ceph-mon[34945]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:29 smithi160 ceph-mon[34696]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:29 smithi039 ceph-mon[30415]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:31 smithi150 ceph-mon[34945]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:31 smithi160 ceph-mon[34696]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:31 smithi039 ceph-mon[30415]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:33 smithi150 ceph-mon[34945]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:33 smithi160 ceph-mon[34696]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:33 smithi039 ceph-mon[30415]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:35.694 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:35 smithi160 ceph-mon[34696]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:35 smithi150 ceph-mon[34945]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:35 smithi039 ceph-mon[30415]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:36.685 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:36 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T00:56:36.685 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:36 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:36.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:36 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T00:56:36.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:36 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:36 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T00:56:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:36 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:37.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:37 smithi160 ceph-mon[34696]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:37.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:37 smithi160 ceph-mon[34696]: Deploying daemon osd.5 on smithi160 2024-09-21T00:56:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:37 smithi150 ceph-mon[34945]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:37 smithi150 ceph-mon[34945]: Deploying daemon osd.5 on smithi160 2024-09-21T00:56:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:37 smithi039 ceph-mon[30415]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:37 smithi039 ceph-mon[30415]: Deploying daemon osd.5 on smithi160 2024-09-21T00:56:39.723 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:39 smithi160 ceph-mon[34696]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:39 smithi039 ceph-mon[30415]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:39 smithi150 ceph-mon[34945]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:41.736 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:41 smithi160 ceph-mon[34696]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:41 smithi039 ceph-mon[30415]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:41 smithi150 ceph-mon[34945]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:42.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:42.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:42.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:42.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:43.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:43.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:43.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:43.746 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:43 smithi160 ceph-mon[34696]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:43 smithi039 ceph-mon[30415]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:43 smithi150 ceph-mon[34945]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:45 smithi160 ceph-mon[34696]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:45 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:45 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:45 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:45 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:45 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:45.657 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 5 on host 'smithi160' 2024-09-21T00:56:45.658 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.657+0000 7f155b7fe640 1 -- 172.21.15.160:0/1868099535 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f156c063800 con 0x7f15540b66b0 2024-09-21T00:56:45.660 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15540b66b0 msgr2=0x7f15540b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:45.660 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15540b66b0 0x7f15540b8bc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f156402a930 tx=0x7f1564005fd0 comp rx=0 tx=0).stop 2024-09-21T00:56:45.660 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 msgr2=0x7f156c19fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:45.660 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c19fd90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f155c00e970 tx=0x7f155c00ed70 comp rx=0 tx=0).stop 2024-09-21T00:56:45.660 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 shutdown_connections 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15540b66b0 0x7f15540b8bc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f156c108d00 0x7f156c19fd90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f156c1083b0 0x7f156c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 --2- 172.21.15.160:0/1868099535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f156c103140 0x7f156c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 >> 172.21.15.160:0/1868099535 conn(0x7f156c077c60 msgr2=0x7f156c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 shutdown_connections 2024-09-21T00:56:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:56:45.659+0000 7f1570a55640 1 -- 172.21.15.160:0/1868099535 wait complete. 2024-09-21T00:56:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:45 smithi039 ceph-mon[30415]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:45 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:45 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:45 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:45 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:45.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:45 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:45 smithi150 ceph-mon[34945]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:45 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:45 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:45 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:45 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:45 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.306 DEBUG:teuthology.orchestra.run.smithi160:osd.5> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.5.service 2024-09-21T00:56:46.309 INFO:tasks.cephadm:Deploying osd.6 on smithi039 with /dev/vg_nvme/lv_2... 2024-09-21T00:56:46.309 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T00:56:46.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:46 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:46.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:46 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:46 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.694 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:56:46.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:46 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:46.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:46 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:46.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:46 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:47.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:46 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:56:47.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:46 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:47.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:46 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:47 smithi160 ceph-mon[34696]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:47 smithi039 ceph-mon[30415]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:47.965 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:47 smithi150 ceph-mon[34945]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:49.024 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 00:56:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T00:56:48.703+0000 7ff655e16640 -1 osd.5 0 log_to_monitors true 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:49.810 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:49.810 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:56:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:56:50.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:56:50.641 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:56:50.675 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi039:vg_nvme/lv_2 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi160 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: from='osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:56:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:50 smithi039 ceph-mon[30415]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:56:51.051 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:56:51.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi160 2024-09-21T00:56:51.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-21T00:56:51.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T00:56:51.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T00:56:51.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:51.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: from='osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:56:51.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:50 smithi150 ceph-mon[34945]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:56:51.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi160 2024-09-21T00:56:51.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-21T00:56:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T00:56:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T00:56:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: from='osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:56:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:50 smithi160 ceph-mon[34696]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:56:51.606 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T00:56:51.303+0000 7ff651d98640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T00:56:51.607 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T00:56:51.309+0000 7ff649b8e640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:56:51.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:51 smithi039 ceph-mon[30415]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:51.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:51 smithi039 ceph-mon[30415]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:56:51.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:51 smithi039 ceph-mon[30415]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T00:56:51.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:51 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:51.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:51 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:51.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:51 smithi039 ceph-mon[30415]: from='osd.5 ' entity='osd.5' 2024-09-21T00:56:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 -- 172.21.15.39:0/3296067434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 msgr2=0x7ff8e4104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/3296067434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e4104dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009900 tx=0x7ff8d402a3f0 comp rx=0 tx=0).stop 2024-09-21T00:56:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 -- 172.21.15.39:0/3296067434 shutdown_connections 2024-09-21T00:56:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/3296067434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8e410de20 0x7ff8e4110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/3296067434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8e4105310 0x7ff8e410d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:51.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/3296067434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e4104dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:51.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 -- 172.21.15.39:0/3296067434 >> 172.21.15.39:0/3296067434 conn(0x7ff8e4079730 msgr2=0x7ff8e4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:56:51.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 -- 172.21.15.39:0/3296067434 shutdown_connections 2024-09-21T00:56:51.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.930+0000 7ff8e8e00640 1 -- 172.21.15.39:0/3296067434 wait complete. 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.931+0000 7ff8e8e00640 1 Processor -- start 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 -- start start 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8e4105310 0x7ff8e419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8e410de20 0x7ff8e41a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4113a80 con 0x7ff8e41049c0 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4113900 con 0x7ff8e410de20 2024-09-21T00:56:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e8e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4113c00 con 0x7ff8e4105310 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43666/0 (socket says 172.21.15.39:43666) 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e1400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8e4105310 0x7ff8e419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e1e00640 1 -- 172.21.15.39:0/236323240 learned_addr learned my addr 172.21.15.39:0/236323240 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.932+0000 7ff8e2800640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8e410de20 0x7ff8e41a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8e1e00640 1 -- 172.21.15.39:0/236323240 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8e4105310 msgr2=0x7ff8e419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8e1e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8e4105310 0x7ff8e419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8e1e00640 1 -- 172.21.15.39:0/236323240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8e410de20 msgr2=0x7ff8e41a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8e1e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8e410de20 0x7ff8e41a1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:56:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8e1e00640 1 -- 172.21.15.39:0/236323240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e41a26f0 con 0x7ff8e41049c0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8e1e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e419c780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4032fd0 tx=0x7ff8d4031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.933+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d403f030 con 0x7ff8e41049c0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.934+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d40043d0 con 0x7ff8e41049c0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.934+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e41a2980 con 0x7ff8e41049c0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.934+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4035070 con 0x7ff8e41049c0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.934+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a9140 con 0x7ff8e41049c0 2024-09-21T00:56:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.935+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e406c1e0 con 0x7ff8e41049c0 2024-09-21T00:56:51.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.937+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff8d4042030 con 0x7ff8e41049c0 2024-09-21T00:56:51.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.939+0000 7ff8daa00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8c80b65e0 0x7ff8c80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:56:51.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.939+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4781+0+0 (secure 0 0 0) 0x7ff8d41068c0 con 0x7ff8e41049c0 2024-09-21T00:56:51.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.939+0000 7ff8e1400640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8c80b65e0 0x7ff8c80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:56:51.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.940+0000 7ff8e1400640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8c80b65e0 0x7ff8c80b8af0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00095f0 tx=0x7ff8d0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:56:51.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:51.943+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff8d40cde80 con 0x7ff8e41049c0 2024-09-21T00:56:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:51 smithi150 ceph-mon[34945]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:51 smithi150 ceph-mon[34945]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:56:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:51 smithi150 ceph-mon[34945]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T00:56:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:51 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:51 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:51 smithi150 ceph-mon[34945]: from='osd.5 ' entity='osd.5' 2024-09-21T00:56:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-mon[34696]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-21T00:56:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-mon[34696]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:56:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-mon[34696]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T00:56:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:51 smithi160 ceph-mon[34696]: from='osd.5 ' entity='osd.5' 2024-09-21T00:56:52.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:56:52.151+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff8e4100d80 con 0x7ff8c80b65e0 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] boot 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:56:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:52.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:52.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:52.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] boot 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:53.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: osd.5 [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] boot 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:56:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:56:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:53 smithi039 ceph-mon[30415]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T00:56:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:53 smithi150 ceph-mon[34945]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T00:56:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:53 smithi160 ceph-mon[34696]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T00:56:54.867 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:54 smithi039 ceph-mon[30415]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T00:56:54.867 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:54 smithi039 ceph-mon[30415]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:56:55.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:54 smithi150 ceph-mon[34945]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T00:56:55.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:54 smithi150 ceph-mon[34945]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:56:55.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:54 smithi160 ceph-mon[34696]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T00:56:55.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:54 smithi160 ceph-mon[34696]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:56:57.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:56 smithi039 ceph-mon[30415]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-21T00:56:57.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4131206972' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2e0496c7-05ba-40e2-80fe-18ded790693a"} : dispatch 2024-09-21T00:56:57.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4131206972' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e0496c7-05ba-40e2-80fe-18ded790693a"}]': finished 2024-09-21T00:56:57.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:56 smithi039 ceph-mon[30415]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T00:56:57.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:56:57.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/459953930' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:56 smithi150 ceph-mon[34945]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-21T00:56:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4131206972' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2e0496c7-05ba-40e2-80fe-18ded790693a"} : dispatch 2024-09-21T00:56:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4131206972' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e0496c7-05ba-40e2-80fe-18ded790693a"}]': finished 2024-09-21T00:56:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:56 smithi150 ceph-mon[34945]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T00:56:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:56:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/459953930' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:57.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:56 smithi160 ceph-mon[34696]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-21T00:56:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4131206972' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2e0496c7-05ba-40e2-80fe-18ded790693a"} : dispatch 2024-09-21T00:56:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4131206972' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e0496c7-05ba-40e2-80fe-18ded790693a"}]': finished 2024-09-21T00:56:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:56 smithi160 ceph-mon[34696]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T00:56:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:56:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/459953930' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:56:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:56:58 smithi150 ceph-mon[34945]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T00:56:59.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:56:58 smithi039 ceph-mon[30415]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T00:56:59.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:56:58 smithi160 ceph-mon[34696]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T00:57:01.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:00 smithi039 ceph-mon[30415]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T00:57:01.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:00 smithi150 ceph-mon[34945]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T00:57:01.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:00 smithi160 ceph-mon[34696]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T00:57:03.000 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:02 smithi039 ceph-mon[30415]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T00:57:03.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:02 smithi150 ceph-mon[34945]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T00:57:03.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:02 smithi160 ceph-mon[34696]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T00:57:05.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:04 smithi039 ceph-mon[30415]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:57:05.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:04 smithi150 ceph-mon[34945]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:57:05.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:04 smithi160 ceph-mon[34696]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:57:07.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:06 smithi039 ceph-mon[30415]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:07.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:06 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T00:57:07.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:06 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:07.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:06 smithi150 ceph-mon[34945]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:07.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:06 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T00:57:07.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:06 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:07.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:06 smithi160 ceph-mon[34696]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:07.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:06 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T00:57:07.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:06 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:07 smithi150 ceph-mon[34945]: Deploying daemon osd.6 on smithi039 2024-09-21T00:57:08.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:07 smithi160 ceph-mon[34696]: Deploying daemon osd.6 on smithi039 2024-09-21T00:57:08.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:07 smithi039 ceph-mon[30415]: Deploying daemon osd.6 on smithi039 2024-09-21T00:57:09.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:08 smithi150 ceph-mon[34945]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:09.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:08 smithi160 ceph-mon[34696]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:09.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:08 smithi039 ceph-mon[30415]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:11.226 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:10 smithi039 ceph-mon[30415]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:11.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:10 smithi150 ceph-mon[34945]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:11.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:10 smithi160 ceph-mon[34696]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:13.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:12 smithi039 ceph-mon[30415]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:13.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:12 smithi150 ceph-mon[34945]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:13.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:12 smithi160 ceph-mon[34696]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:15.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:14 smithi039 ceph-mon[30415]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:14 smithi150 ceph-mon[34945]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:15.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:14 smithi160 ceph-mon[34696]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:17.055 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:16 smithi039 ceph-mon[30415]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:16 smithi150 ceph-mon[34945]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:17.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:16 smithi160 ceph-mon[34696]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:18.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:18.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:18.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:19.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:19 smithi039 ceph-mon[30415]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:19.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:19 smithi150 ceph-mon[34945]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:19.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:19 smithi160 ceph-mon[34696]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:21 smithi160 ceph-mon[34696]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:21 smithi039 ceph-mon[30415]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:21 smithi150 ceph-mon[34945]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.197+0000 7ff8daa00640 1 -- 172.21.15.39:0/236323240 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff8e4100d80 con 0x7ff8c80b65e0 2024-09-21T00:57:23.198 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 6 on host 'smithi039' 2024-09-21T00:57:23.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8c80b65e0 msgr2=0x7ff8c80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:23.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8c80b65e0 0x7ff8c80b8af0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00095f0 tx=0x7ff8d0005f90 comp rx=0 tx=0).stop 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 msgr2=0x7ff8e419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e419c780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4032fd0 tx=0x7ff8d4031a20 comp rx=0 tx=0).stop 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 shutdown_connections 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8c80b65e0 0x7ff8c80b8af0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8e410de20 0x7ff8e41a1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8e4105310 0x7ff8e419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 --2- 172.21.15.39:0/236323240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e41049c0 0x7ff8e419c780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 >> 172.21.15.39:0/236323240 conn(0x7ff8e4079730 msgr2=0x7ff8e4100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 shutdown_connections 2024-09-21T00:57:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:57:23.200+0000 7ff8e8e00640 1 -- 172.21.15.39:0/236323240 wait complete. 2024-09-21T00:57:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:23 smithi160 ceph-mon[34696]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:23 smithi160 ceph-mon[34696]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T00:57:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:23 smithi039 ceph-mon[30415]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:23 smithi039 ceph-mon[30415]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T00:57:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:23 smithi150 ceph-mon[34945]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:23 smithi150 ceph-mon[34945]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T00:57:24.264 DEBUG:teuthology.orchestra.run.smithi039:osd.6> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.6.service 2024-09-21T00:57:24.267 INFO:tasks.cephadm:Deploying osd.7 on smithi150 with /dev/vg_nvme/lv_2... 2024-09-21T00:57:24.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T00:57:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:57:25.473 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:25 smithi150 ceph-mon[34945]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:25.473 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:25 smithi150 ceph-mon[34945]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T00:57:25.473 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:25 smithi150 ceph-mon[34945]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T00:57:25.473 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:25 smithi150 ceph-mon[34945]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:57:25.473 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:25.521 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-mon[30415]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:25.521 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-mon[30415]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T00:57:25.522 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-mon[30415]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T00:57:25.522 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-mon[30415]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:57:25.522 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:25 smithi160 ceph-mon[34696]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:25 smithi160 ceph-mon[34696]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:25 smithi160 ceph-mon[34696]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:25 smithi160 ceph-mon[34696]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T00:57:25.889+0000 7efe16800640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T00:57:26.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 00:57:25 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T00:57:25.909+0000 7efe0c800640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:57:26.475 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:26 smithi039 ceph-mon[30415]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:57:26.475 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:26 smithi039 ceph-mon[30415]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T00:57:26.475 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.475 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.475 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:26 smithi039 ceph-mon[30415]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' 2024-09-21T00:57:26.494 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:26 smithi150 ceph-mon[34945]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:57:26.494 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:26 smithi150 ceph-mon[34945]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T00:57:26.494 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.495 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.495 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:26 smithi150 ceph-mon[34945]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' 2024-09-21T00:57:26.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:26 smithi160 ceph-mon[34696]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:57:26.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:26 smithi160 ceph-mon[34696]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T00:57:26.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:26.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:26 smithi160 ceph-mon[34696]: from='osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111]' entity='osd.6' 2024-09-21T00:57:27.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:57:27.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:57:27.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:27.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:27.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] boot 2024-09-21T00:57:27.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T00:57:27.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:27.369 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:57:27.398 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] boot 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:27.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:57:27.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:57:27.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T00:57:27.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:27.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: osd.6 [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] boot 2024-09-21T00:57:27.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T00:57:27.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T00:57:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.172+0000 7f44c89ba640 1 -- 172.21.15.150:0/3835956511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c4102ae0 msgr2=0x7f44c4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.172+0000 7f44c89ba640 1 --2- 172.21.15.150:0/3835956511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c4102ae0 0x7f44c4102ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009900 tx=0x7f44bc02a410 comp rx=0 tx=0).stop 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 -- 172.21.15.150:0/3835956511 shutdown_connections 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 --2- 172.21.15.150:0/3835956511 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44c4108d20 0x7f44c410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 --2- 172.21.15.150:0/3835956511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c41083d0 0x7f44c41087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 --2- 172.21.15.150:0/3835956511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c4102ae0 0x7f44c4102ef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 -- 172.21.15.150:0/3835956511 >> 172.21.15.150:0/3835956511 conn(0x7f44c40797b0 msgr2=0x7f44c4079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 -- 172.21.15.150:0/3835956511 shutdown_connections 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 -- 172.21.15.150:0/3835956511 wait complete. 2024-09-21T00:57:28.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.173+0000 7f44c89ba640 1 Processor -- start 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 -- start start 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44c4102ae0 0x7f44c41993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 0x7f44c4199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c4108d20 0x7f44c419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f44c41129e0 con 0x7f44c4108d20 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f44c4112860 con 0x7f44c41083d0 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c89ba640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f44c4112b60 con 0x7f44c4102ae0 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 0x7f44c4199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 0x7f44c4199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57908/0 (socket says 172.21.15.150:57908) 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 -- 172.21.15.150:0/1448300597 learned_addr learned my addr 172.21.15.150:0/1448300597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c2d9d640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c4108d20 0x7f44c419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c259c640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44c4102ae0 0x7f44c41993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 -- 172.21.15.150:0/1448300597 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44c4102ae0 msgr2=0x7f44c41993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44c4102ae0 0x7f44c41993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 -- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c4108d20 msgr2=0x7f44c419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c4108d20 0x7f44c419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.174+0000 7f44c1d9b640 1 -- 172.21.15.150:0/1448300597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c419f320 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.175+0000 7f44c1d9b640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 0x7f44c4199950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44b800ea50 tx=0x7f44b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.175+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b800cdc0 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.175+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c419f610 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.175+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b8011920 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.175+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b801aa60 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.175+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c41a5d70 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.176+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44c4102ef0 con 0x7f44c41083d0 2024-09-21T00:57:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.178+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f44b8011ad0 con 0x7f44c41083d0 2024-09-21T00:57:28.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.179+0000 7f44b37fe640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f44ac0b6560 0x7f44ac0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:28.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.179+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5180+0+0 (secure 0 0 0) 0x7f44b80e6e90 con 0x7f44c41083d0 2024-09-21T00:57:28.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.179+0000 7f44c259c640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f44ac0b6560 0x7f44ac0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:28.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.180+0000 7f44c259c640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f44ac0b6560 0x7f44ac0b8a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44bc031440 tx=0x7f44bc005d50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:57:28.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.182+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f44b8058710 con 0x7f44c41083d0 2024-09-21T00:57:28.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:28.378+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f44c4063800 con 0x7f44ac0b6560 2024-09-21T00:57:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:29.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.509 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:29.509 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:29.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:29.511 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:29.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:29.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:29.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:29.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:30.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:30 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi039 2024-09-21T00:57:30.510 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:30 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 5696M 2024-09-21T00:57:30.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:30 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi039 2024-09-21T00:57:30.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:30 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 5696M 2024-09-21T00:57:30.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:30 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi039 2024-09-21T00:57:30.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:30 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 5696M 2024-09-21T00:57:31.251 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:31 smithi039 ceph-mon[30415]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:31.251 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:31 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/1779728210' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"} : dispatch 2024-09-21T00:57:31.251 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:31 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"} : dispatch 2024-09-21T00:57:31.251 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:31 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"}]': finished 2024-09-21T00:57:31.252 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:31 smithi039 ceph-mon[30415]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T00:57:31.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:31 smithi150 ceph-mon[34945]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:31.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:31 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/1779728210' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"} : dispatch 2024-09-21T00:57:31.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:31 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"} : dispatch 2024-09-21T00:57:31.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:31 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"}]': finished 2024-09-21T00:57:31.476 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:31 smithi150 ceph-mon[34945]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T00:57:31.476 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:31.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:31 smithi160 ceph-mon[34696]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:31.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:31 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/1779728210' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"} : dispatch 2024-09-21T00:57:31.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:31 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"} : dispatch 2024-09-21T00:57:31.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:31 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec370951-5b96-470f-ad4d-1122615da035"}]': finished 2024-09-21T00:57:31.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:31 smithi160 ceph-mon[34696]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T00:57:31.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:32 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/2428809504' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:57:32.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:32 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/2428809504' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:57:32.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:32 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/2428809504' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:57:33.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:33 smithi150 ceph-mon[34945]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:33.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:33 smithi160 ceph-mon[34696]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:33.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:33 smithi039 ceph-mon[30415]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:35 smithi150 ceph-mon[34945]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:35 smithi160 ceph-mon[34696]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:35 smithi039 ceph-mon[30415]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:37.457 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:37 smithi150 ceph-mon[34945]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:37.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:37 smithi160 ceph-mon[34696]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:37.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:37 smithi039 ceph-mon[30415]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:39.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:39 smithi150 ceph-mon[34945]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:39.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:39 smithi160 ceph-mon[34696]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:39 smithi039 ceph-mon[30415]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:40.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:40 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T00:57:40.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:40 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:40.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:40 smithi150 ceph-mon[34945]: Deploying daemon osd.7 on smithi150 2024-09-21T00:57:40.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:40 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T00:57:40.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:40 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:40.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:40 smithi160 ceph-mon[34696]: Deploying daemon osd.7 on smithi150 2024-09-21T00:57:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:40 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T00:57:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:40 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:40 smithi039 ceph-mon[30415]: Deploying daemon osd.7 on smithi150 2024-09-21T00:57:41.477 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:41 smithi150 ceph-mon[34945]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:41 smithi160 ceph-mon[34696]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:41 smithi039 ceph-mon[30415]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:43.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:43 smithi150 ceph-mon[34945]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:43.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:43 smithi160 ceph-mon[34696]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:43.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:43 smithi039 ceph-mon[30415]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:45.459 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:45 smithi150 ceph-mon[34945]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:45 smithi160 ceph-mon[34696]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:45.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:45 smithi039 ceph-mon[30415]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:46.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:46 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:46.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:46 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:46.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:46 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:46.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:46 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:46.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:46 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:46.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:46 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:46.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:46 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:46.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:46 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:46.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:46 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:47.451 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:47 smithi150 ceph-mon[34945]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:47.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:47 smithi160 ceph-mon[34696]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:47.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:47 smithi039 ceph-mon[30415]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:49.430 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:49 smithi150 ceph-mon[34945]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:49.430 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.430 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.430 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:49.431 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:49.431 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:49 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:49 smithi160 ceph-mon[34696]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:49.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:49.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:49.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:49 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:49 smithi039 ceph-mon[30415]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:49 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:49.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.847+0000 7f44b37fe640 1 -- 172.21.15.150:0/1448300597 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f44c4063800 con 0x7f44ac0b6560 2024-09-21T00:57:49.848 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 7 on host 'smithi150' 2024-09-21T00:57:49.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f44ac0b6560 msgr2=0x7f44ac0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:49.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f44ac0b6560 0x7f44ac0b8a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44bc031440 tx=0x7f44bc005d50 comp rx=0 tx=0).stop 2024-09-21T00:57:49.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 msgr2=0x7f44c4199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:49.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 0x7f44c4199950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44b800ea50 tx=0x7f44b800ef10 comp rx=0 tx=0).stop 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 shutdown_connections 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c4108d20 0x7f44c419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f44ac0b6560 0x7f44ac0b8a70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c41083d0 0x7f44c4199950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 --2- 172.21.15.150:0/1448300597 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44c4102ae0 0x7f44c41993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.849+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 >> 172.21.15.150:0/1448300597 conn(0x7f44c40797b0 msgr2=0x7f44c4106460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.850+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 shutdown_connections 2024-09-21T00:57:49.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:57:49.850+0000 7f44c89ba640 1 -- 172.21.15.150:0/1448300597 wait complete. 2024-09-21T00:57:50.487 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:50.487 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:50.487 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:50.517 DEBUG:teuthology.orchestra.run.smithi150:osd.7> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.7.service 2024-09-21T00:57:50.520 INFO:tasks.cephadm:Deploying osd.8 on smithi160 with /dev/vg_nvme/lv_2... 2024-09-21T00:57:50.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T00:57:50.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:50.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:50.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:50.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:57:50.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:50.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:50.767 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:57:51.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:51 smithi160 ceph-mon[34696]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:51.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:51 smithi039 ceph-mon[30415]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:51.677 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:51 smithi150 ceph-mon[34945]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:52.017 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 00:57:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T00:57:51.784+0000 7f2281d76640 -1 osd.7 0 log_to_monitors true 2024-09-21T00:57:52.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:52 smithi160 ceph-mon[34696]: from='osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T00:57:52.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:52 smithi160 ceph-mon[34696]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T00:57:52.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:52 smithi039 ceph-mon[30415]: from='osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T00:57:52.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:52 smithi039 ceph-mon[30415]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T00:57:52.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:52 smithi150 ceph-mon[34945]: from='osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T00:57:52.780 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:52 smithi150 ceph-mon[34945]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T00:57:53.412 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:57:53.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_2 2024-09-21T00:57:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:53 smithi039 ceph-mon[30415]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:53 smithi039 ceph-mon[30415]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T00:57:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:53 smithi039 ceph-mon[30415]: from='osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:57:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:53 smithi039 ceph-mon[30415]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T00:57:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:53 smithi039 ceph-mon[30415]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:57:53.687 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:57:53.802 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:53 smithi160 ceph-mon[34696]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:53.802 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:53 smithi160 ceph-mon[34696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T00:57:53.802 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:53 smithi160 ceph-mon[34696]: from='osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:57:53.802 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:53 smithi160 ceph-mon[34696]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T00:57:53.802 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:53.802 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:53 smithi160 ceph-mon[34696]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:57:53.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:53 smithi150 ceph-mon[34945]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:53 smithi150 ceph-mon[34945]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T00:57:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:53 smithi150 ceph-mon[34945]: from='osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:57:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:53 smithi150 ceph-mon[34945]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T00:57:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:53 smithi150 ceph-mon[34945]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1661573698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 msgr2=0x7fa9d4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1661573698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4102e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0009900 tx=0x7fa9d002a410 comp rx=0 tx=0).stop 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1661573698 shutdown_connections 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1661573698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d4108cf0 0x7fa9d410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1661573698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 0x7fa9d41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1661573698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4102e70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:54.148 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1661573698 >> 172.21.15.160:0/1661573698 conn(0x7fa9d4079760 msgr2=0x7fa9d4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:57:54.149 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1661573698 shutdown_connections 2024-09-21T00:57:54.149 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.147+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1661573698 wait complete. 2024-09-21T00:57:54.149 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 Processor -- start 2024-09-21T00:57:54.149 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 -- start start 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 0x7fa9d4198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d4108cf0 0x7fa9d419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d41118b0 con 0x7fa9d41083a0 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d4111730 con 0x7fa9d4108cf0 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9dba4b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d4111a30 con 0x7fa9d4102a60 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:52158/0 (socket says 172.21.15.160:52158) 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 -- 172.21.15.160:0/1100605966 learned_addr learned my addr 172.21.15.160:0/1100605966 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-21T00:57:54.151 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d9fe8640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d4108cf0 0x7fa9d419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d8fe6640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 0x7fa9d4198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 -- 172.21.15.160:0/1100605966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d4108cf0 msgr2=0x7fa9d419dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d4108cf0 0x7fa9d419dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 -- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 msgr2=0x7fa9d4198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 0x7fa9d4198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d97e7640 1 -- 172.21.15.160:0/1100605966 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d419e210 con 0x7fa9d4102a60 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.148+0000 7fa9d8fe6640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 0x7fa9d4198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.149+0000 7fa9d97e7640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4198360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0005940 tx=0x7fa9d0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.149+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d003f030 con 0x7fa9d4102a60 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.149+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9d0003aa0 con 0x7fa9d4102a60 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.149+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0035070 con 0x7fa9d4102a60 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.149+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d419e4a0 con 0x7fa9d4102a60 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.149+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d419e930 con 0x7fa9d4102a60 2024-09-21T00:57:54.152 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.150+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d4103ce0 con 0x7fa9d4102a60 2024-09-21T00:57:54.153 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.152+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fa9d0042030 con 0x7fa9d4102a60 2024-09-21T00:57:54.153 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.153+0000 7fa9ca7fc640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9c00b6590 0x7fa9c00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:57:54.153 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.153+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5332+0+0 (secure 0 0 0) 0x7fa9d0107240 con 0x7fa9d4102a60 2024-09-21T00:57:54.153 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.153+0000 7fa9d8fe6640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9c00b6590 0x7fa9c00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:57:54.154 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.154+0000 7fa9d8fe6640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9c00b6590 0x7fa9c00b8aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9d406b230 tx=0x7fa9c400a5c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:57:54.157 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.156+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa9d00ce5d0 con 0x7fa9d4102a60 2024-09-21T00:57:54.314 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T00:57:54.014+0000 7f227e50d640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T00:57:54.314 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T00:57:54.023+0000 7f2275aee640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:57:54.345 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:57:54.344+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa9d410ab20 con 0x7fa9c00b6590 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi150 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi150 to 5696M 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:54.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='osd.7 ' entity='osd.7' 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] boot 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:57:54.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:54.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:57:54.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T00:57:54.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi150 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi150 to 5696M 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='osd.7 ' entity='osd.7' 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] boot 2024-09-21T00:57:54.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T00:57:54.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:54.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:57:54.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:57:54.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:54.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:57:54.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi150 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi150 to 5696M 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='osd.7 ' entity='osd.7' 2024-09-21T00:57:54.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: osd.7 [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] boot 2024-09-21T00:57:54.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T00:57:54.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T00:57:54.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:57:54.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:57:54.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:57:55.812 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:55 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:57:55.812 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:55 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:57:55.812 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:55 smithi160 ceph-mon[34696]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:55.812 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:55 smithi160 ceph-mon[34696]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:55.812 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:55 smithi160 ceph-mon[34696]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T00:57:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:55 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:57:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:55 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:57:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:55 smithi039 ceph-mon[30415]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:55 smithi039 ceph-mon[30415]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:55 smithi039 ceph-mon[30415]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T00:57:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:55 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:57:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:55 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:57:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:55 smithi150 ceph-mon[34945]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-21T00:57:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:55 smithi150 ceph-mon[34945]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:55 smithi150 ceph-mon[34945]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/988097135' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"} : dispatch 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"} : dispatch 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"}]': finished 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T00:57:57.535 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/988097135' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"} : dispatch 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"} : dispatch 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"}]': finished 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T00:57:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:57:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:57:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T00:57:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/988097135' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"} : dispatch 2024-09-21T00:57:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"} : dispatch 2024-09-21T00:57:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "289d3eda-511f-4731-9242-02e987aafefd"}]': finished 2024-09-21T00:57:57.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T00:57:57.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:57:58.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:58 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/1854219843' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:57:58.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:58 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/1854219843' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:57:58.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:58 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/1854219843' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:57:59.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:57:59 smithi150 ceph-mon[34945]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:57:59.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:57:59 smithi160 ceph-mon[34696]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:57:59.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:57:59 smithi039 ceph-mon[30415]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:01.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:01 smithi039 ceph-mon[30415]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-09-21T00:58:01.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:01 smithi150 ceph-mon[34945]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-09-21T00:58:01.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:01 smithi160 ceph-mon[34696]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-09-21T00:58:03.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:03 smithi160 ceph-mon[34696]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T00:58:03.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:03 smithi150 ceph-mon[34945]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T00:58:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:03 smithi039 ceph-mon[30415]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T00:58:05.492 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:05 smithi160 ceph-mon[34696]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T00:58:05.493 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T00:58:05.493 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:05.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:05 smithi150 ceph-mon[34945]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T00:58:05.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T00:58:05.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:05 smithi039 ceph-mon[30415]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T00:58:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T00:58:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:06.478 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:06 smithi160 ceph-mon[34696]: Deploying daemon osd.8 on smithi160 2024-09-21T00:58:06.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:06 smithi150 ceph-mon[34945]: Deploying daemon osd.8 on smithi160 2024-09-21T00:58:06.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:06 smithi039 ceph-mon[30415]: Deploying daemon osd.8 on smithi160 2024-09-21T00:58:07.477 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:07 smithi160 ceph-mon[34696]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T00:58:07.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:07 smithi150 ceph-mon[34945]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T00:58:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:07 smithi039 ceph-mon[30415]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T00:58:09.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:09 smithi150 ceph-mon[34945]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T00:58:09.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:09 smithi160 ceph-mon[34696]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T00:58:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:09 smithi039 ceph-mon[30415]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T00:58:11.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:11 smithi150 ceph-mon[34945]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:58:11.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:11 smithi160 ceph-mon[34696]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:58:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:11 smithi039 ceph-mon[30415]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:58:12.521 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:12.522 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:12.522 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:12.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:12.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:12.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:12.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:12.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:12.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:13.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:13 smithi150 ceph-mon[34945]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:13.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:13 smithi160 ceph-mon[34696]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:13 smithi039 ceph-mon[30415]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:15.015 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.014+0000 7fa9ca7fc640 1 -- 172.21.15.160:0/1100605966 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa9d410ab20 con 0x7fa9c00b6590 2024-09-21T00:58:15.016 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 8 on host 'smithi160' 2024-09-21T00:58:15.017 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.016+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9c00b6590 msgr2=0x7fa9c00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:15.017 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.016+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9c00b6590 0x7fa9c00b8aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9d406b230 tx=0x7fa9c400a5c0 comp rx=0 tx=0).stop 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 msgr2=0x7fa9d4198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4198360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0005940 tx=0x7fa9d0004270 comp rx=0 tx=0).stop 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 shutdown_connections 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9c00b6590 0x7fa9c00b8aa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d4108cf0 0x7fa9d419dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d41083a0 0x7fa9d4198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 --2- 172.21.15.160:0/1100605966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9d4102a60 0x7fa9d4198360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 >> 172.21.15.160:0/1100605966 conn(0x7fa9d4079760 msgr2=0x7fa9d41064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 shutdown_connections 2024-09-21T00:58:15.018 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:58:15.017+0000 7fa9dba4b640 1 -- 172.21.15.160:0/1100605966 wait complete. 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.294 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:15.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:15.722 DEBUG:teuthology.orchestra.run.smithi160:osd.8> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.8.service 2024-09-21T00:58:15.725 INFO:tasks.cephadm:Deploying osd.9 on smithi039 with /dev/vg_nvme/lv_1... 2024-09-21T00:58:15.725 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T00:58:16.112 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:58:17.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:17 smithi160 ceph-mon[34696]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:17.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:17 smithi150 ceph-mon[34945]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:17.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:17 smithi039 ceph-mon[30415]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:18.027 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 00:58:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T00:58:17.654+0000 7f6024ba5640 -1 osd.8 0 log_to_monitors true 2024-09-21T00:58:18.211 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:18 smithi039 ceph-mon[30415]: from='osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T00:58:18.211 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:18 smithi039 ceph-mon[30415]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T00:58:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:18 smithi150 ceph-mon[34945]: from='osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T00:58:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:18 smithi150 ceph-mon[34945]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T00:58:18.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:18 smithi160 ceph-mon[34696]: from='osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T00:58:18.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:18 smithi160 ceph-mon[34696]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T00:58:19.049 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:19.049 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T00:58:19.049 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T00:58:19.049 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:19.050 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:58:19.050 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:58:19.050 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi160 2024-09-21T00:58:19.050 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.050 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.050 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.051 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.051 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.051 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-21T00:58:19.051 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.051 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:19.051 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:19.052 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:58:19.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi160 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:19.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.357 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T00:58:19.046+0000 7f6020b27640 -1 osd.8 0 waiting for initial osdmap 2024-09-21T00:58:19.357 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 00:58:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T00:58:19.062+0000 7f601c925640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:58:19.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:19.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T00:58:19.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T00:58:19.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:19.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:58:19.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:58:19.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi160 2024-09-21T00:58:19.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:19.405 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-21T00:58:19.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:19.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:19.406 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:19.656 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:58:19.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi039:vg_nvme/lv_1 2024-09-21T00:58:20.073 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:58:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:58:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: osdmap e53: 9 total, 8 up, 9 in 2024-09-21T00:58:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] boot 2024-09-21T00:58:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T00:58:20.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:58:20.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: osdmap e53: 9 total, 8 up, 9 in 2024-09-21T00:58:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] boot 2024-09-21T00:58:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T00:58:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: osdmap e53: 9 total, 8 up, 9 in 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: osd.8 [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] boot 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T00:58:20.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T00:58:20.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 -- 172.21.15.39:0/3492342941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 msgr2=0x7f94d010fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 --2- 172.21.15.39:0/3492342941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d010fc20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009900 tx=0x7f94bc02a410 comp rx=0 tx=0).stop 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 -- 172.21.15.39:0/3492342941 shutdown_connections 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 --2- 172.21.15.39:0/3492342941 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94d0110160 0x7f94d0112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 --2- 172.21.15.39:0/3492342941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d010fc20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 --2- 172.21.15.39:0/3492342941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d0075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 -- 172.21.15.39:0/3492342941 >> 172.21.15.39:0/3492342941 conn(0x7f94d00ff5d0 msgr2=0x7f94d0101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 -- 172.21.15.39:0/3492342941 shutdown_connections 2024-09-21T00:58:20.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.760+0000 7f94d7e00640 1 -- 172.21.15.39:0/3492342941 wait complete. 2024-09-21T00:58:20.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.761+0000 7f94d7e00640 1 Processor -- start 2024-09-21T00:58:20.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.761+0000 7f94d7e00640 1 -- start start 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94d0110160 0x7f94d01a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f94d0115ca0 con 0x7f94d0076080 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94d0115b20 con 0x7f94d0075730 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94d0115e20 con 0x7f94d0110160 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:20.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:51044/0 (socket says 172.21.15.39:51044) 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d5400640 1 -- 172.21.15.39:0/1848162364 learned_addr learned my addr 172.21.15.39:0/1848162364 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.762+0000 7f94d5e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94d0110160 0x7f94d01a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d5400640 1 -- 172.21.15.39:0/1848162364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94d0110160 msgr2=0x7f94d01a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d5400640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94d0110160 0x7f94d01a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d5400640 1 -- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 msgr2=0x7f94d019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d5400640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d5400640 1 -- 172.21.15.39:0/1848162364 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d01a2640 con 0x7f94d0075730 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d4a00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:58:20.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d5400640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d019c6d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94c400d950 tx=0x7f94c400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:58:20.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c4017030 con 0x7f94d0075730 2024-09-21T00:58:20.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d01a2930 con 0x7f94d0075730 2024-09-21T00:58:20.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94c4004020 con 0x7f94d0075730 2024-09-21T00:58:20.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.763+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c40047a0 con 0x7f94d0075730 2024-09-21T00:58:20.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.764+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d01a9090 con 0x7f94d0075730 2024-09-21T00:58:20.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.764+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d0075b40 con 0x7f94d0075730 2024-09-21T00:58:20.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.767+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f94c4009a30 con 0x7f94d0075730 2024-09-21T00:58:20.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.769+0000 7f94ce000640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94b80b65e0 0x7f94b80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:20.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.769+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6044+0+0 (secure 0 0 0) 0x7f94c40e6b30 con 0x7f94d0075730 2024-09-21T00:58:20.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.769+0000 7f94d4a00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94b80b65e0 0x7f94b80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:20.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.770+0000 7f94d4a00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94b80b65e0 0x7f94b80b8af0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009230 tx=0x7f94bc031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:58:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.771+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f94c40adc80 con 0x7f94d0075730 2024-09-21T00:58:20.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:20.981+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f94d010a410 con 0x7f94b80b65e0 2024-09-21T00:58:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:58:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:58:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T00:58:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:58:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:58:21.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:58:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:58:21.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:22.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:22 smithi150 ceph-mon[34945]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:22.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:22 smithi150 ceph-mon[34945]: osdmap e56: 9 total, 9 up, 9 in 2024-09-21T00:58:22.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:22 smithi160 ceph-mon[34696]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:22.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:22 smithi160 ceph-mon[34696]: osdmap e56: 9 total, 9 up, 9 in 2024-09-21T00:58:22.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:22 smithi039 ceph-mon[30415]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:22.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:22 smithi039 ceph-mon[30415]: osdmap e56: 9 total, 9 up, 9 in 2024-09-21T00:58:23.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:23 smithi150 ceph-mon[34945]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:23.571 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:23 smithi039 ceph-mon[30415]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:23.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:23 smithi160 ceph-mon[34696]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:25.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:25 smithi039 ceph-mon[30415]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:25.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:25 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1192132678' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "eb6e99ce-4a27-484a-8850-52b165922de1"} : dispatch 2024-09-21T00:58:25.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:25 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1192132678' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb6e99ce-4a27-484a-8850-52b165922de1"}]': finished 2024-09-21T00:58:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:25 smithi039 ceph-mon[30415]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T00:58:25.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:25 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:25 smithi150 ceph-mon[34945]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:25 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1192132678' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "eb6e99ce-4a27-484a-8850-52b165922de1"} : dispatch 2024-09-21T00:58:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:25 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1192132678' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb6e99ce-4a27-484a-8850-52b165922de1"}]': finished 2024-09-21T00:58:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:25 smithi150 ceph-mon[34945]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T00:58:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:25 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:25.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:25 smithi160 ceph-mon[34696]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:25 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1192132678' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "eb6e99ce-4a27-484a-8850-52b165922de1"} : dispatch 2024-09-21T00:58:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:25 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1192132678' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb6e99ce-4a27-484a-8850-52b165922de1"}]': finished 2024-09-21T00:58:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:25 smithi160 ceph-mon[34696]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T00:58:25.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:25 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:26.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:26 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4084363379' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:58:26.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:26 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4084363379' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:58:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:26 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4084363379' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:58:27.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:27 smithi150 ceph-mon[34945]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T00:58:27.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:27 smithi160 ceph-mon[34696]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T00:58:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:27 smithi039 ceph-mon[30415]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T00:58:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:29 smithi039 ceph-mon[30415]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T00:58:29.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:29 smithi150 ceph-mon[34945]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T00:58:29.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:29 smithi160 ceph-mon[34696]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T00:58:31.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:31 smithi039 ceph-mon[30415]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T00:58:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:31 smithi150 ceph-mon[34945]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T00:58:31.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:31 smithi160 ceph-mon[34696]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T00:58:33.506 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:33 smithi039 ceph-mon[30415]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:58:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:33 smithi150 ceph-mon[34945]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:58:33.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:33 smithi160 ceph-mon[34696]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:58:35.477 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:35 smithi039 ceph-mon[30415]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:58:35.477 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T00:58:35.477 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:35 smithi150 ceph-mon[34945]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:58:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T00:58:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:35 smithi160 ceph-mon[34696]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T00:58:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T00:58:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:36.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:36 smithi150 ceph-mon[34945]: Deploying daemon osd.9 on smithi039 2024-09-21T00:58:36.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:36 smithi160 ceph-mon[34696]: Deploying daemon osd.9 on smithi039 2024-09-21T00:58:36.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:36 smithi039 ceph-mon[30415]: Deploying daemon osd.9 on smithi039 2024-09-21T00:58:37.354 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:37 smithi039 ceph-mon[30415]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:37.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:37 smithi150 ceph-mon[34945]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:37.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:37 smithi160 ceph-mon[34696]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:39.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:39 smithi039 ceph-mon[30415]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:39 smithi150 ceph-mon[34945]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:39 smithi160 ceph-mon[34696]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:41.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:41 smithi150 ceph-mon[34945]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:41.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:41 smithi160 ceph-mon[34696]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:41 smithi039 ceph-mon[30415]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:43.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:43 smithi150 ceph-mon[34945]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:43.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:43 smithi160 ceph-mon[34696]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:43.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:43 smithi039 ceph-mon[30415]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:44.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:44 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:44.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:44 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:44.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:44 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:44.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:44 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:44.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:44 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:44.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:44 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:44.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:44 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:44.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:44 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:44.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:44 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:45 smithi150 ceph-mon[34945]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:45 smithi160 ceph-mon[34696]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:45.863 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:45 smithi039 ceph-mon[30415]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:47 smithi150 ceph-mon[34945]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:47 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:47 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:47 smithi160 ceph-mon[34696]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:47.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:47 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:47.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:47 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:47 smithi039 ceph-mon[30415]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:47 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:47 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.462+0000 7f94ce000640 1 -- 172.21.15.39:0/1848162364 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f94d010a410 con 0x7f94b80b65e0 2024-09-21T00:58:48.464 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 9 on host 'smithi039' 2024-09-21T00:58:48.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94b80b65e0 msgr2=0x7f94b80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:48.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94b80b65e0 0x7f94b80b8af0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009230 tx=0x7f94bc031400 comp rx=0 tx=0).stop 2024-09-21T00:58:48.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 msgr2=0x7f94d019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:48.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d019c6d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94c400d950 tx=0x7f94c400de10 comp rx=0 tx=0).stop 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 shutdown_connections 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94d0110160 0x7f94d01a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94d0076080 0x7f94d019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94b80b65e0 0x7f94b80b8af0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 --2- 172.21.15.39:0/1848162364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94d0075730 0x7f94d019c6d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 >> 172.21.15.39:0/1848162364 conn(0x7f94d00ff5d0 msgr2=0x7f94d0109d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 shutdown_connections 2024-09-21T00:58:48.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:58:48.466+0000 7f94d7e00640 1 -- 172.21.15.39:0/1848162364 wait complete. 2024-09-21T00:58:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:48.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:48.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:58:48.899 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:48.899 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:49.416 DEBUG:teuthology.orchestra.run.smithi039:osd.9> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.9.service 2024-09-21T00:58:49.421 INFO:tasks.cephadm:Deploying osd.10 on smithi150 with /dev/vg_nvme/lv_1... 2024-09-21T00:58:49.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T00:58:49.680 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:58:49.801 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:49 smithi150 ceph-mon[34945]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:49.801 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:49 smithi150 ceph-mon[34945]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T00:58:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:49 smithi039 ceph-mon[30415]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:49 smithi039 ceph-mon[30415]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T00:58:50.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:49 smithi160 ceph-mon[34696]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:50.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:49 smithi160 ceph-mon[34696]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T00:58:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:50 smithi150 ceph-mon[34945]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T00:58:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:50 smithi150 ceph-mon[34945]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T00:58:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:50 smithi150 ceph-mon[34945]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:58:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:50 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:50 smithi150 ceph-mon[34945]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:58:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:50 smithi150 ceph-mon[34945]: osdmap e59: 10 total, 9 up, 10 in 2024-09-21T00:58:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-mon[30415]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T00:58:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-mon[30415]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T00:58:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-mon[30415]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:58:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-mon[30415]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:58:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-mon[30415]: osdmap e59: 10 total, 9 up, 10 in 2024-09-21T00:58:50.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T00:58:50.641+0000 7ff95da00640 -1 osd.9 0 waiting for initial osdmap 2024-09-21T00:58:50.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 00:58:50 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T00:58:50.654+0000 7ff958a00640 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:58:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:50 smithi160 ceph-mon[34696]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T00:58:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:50 smithi160 ceph-mon[34696]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T00:58:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:50 smithi160 ceph-mon[34696]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]} : dispatch 2024-09-21T00:58:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:50 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:50 smithi160 ceph-mon[34696]: from='osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-09-21T00:58:51.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:50 smithi160 ceph-mon[34696]: osdmap e59: 10 total, 9 up, 10 in 2024-09-21T00:58:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:51 smithi039 ceph-mon[30415]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:51 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:51 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:51.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:51 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:51 smithi150 ceph-mon[34945]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:51 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:51 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:51 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:51 smithi160 ceph-mon[34696]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:51 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:51 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:51 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.420 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-21T00:58:52.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2024-09-21T00:58:52.699 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T00:58:52.734 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:52 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:58:52.734 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:52 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:58:52.734 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:52 smithi150 ceph-mon[34945]: osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] boot 2024-09-21T00:58:52.734 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:52 smithi150 ceph-mon[34945]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T00:58:52.734 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:52 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:58:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:52 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:58:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:52 smithi039 ceph-mon[30415]: osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] boot 2024-09-21T00:58:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:52 smithi039 ceph-mon[30415]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T00:58:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:53.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:52 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:58:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:52 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:58:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:52 smithi160 ceph-mon[34696]: osd.9 [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] boot 2024-09-21T00:58:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:52 smithi160 ceph-mon[34696]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T00:58:53.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 -- 172.21.15.150:0/4160324685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe241052b0 msgr2=0x7fbe2410d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 --2- 172.21.15.150:0/4160324685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe241052b0 0x7fbe2410d8f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009900 tx=0x7fbe1802a3f0 comp rx=0 tx=0).stop 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 -- 172.21.15.150:0/4160324685 shutdown_connections 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 --2- 172.21.15.150:0/4160324685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe2410de30 0x7fbe24110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 --2- 172.21.15.150:0/4160324685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe241052b0 0x7fbe2410d8f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 --2- 172.21.15.150:0/4160324685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe24104880 0x7fbe24104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 -- 172.21.15.150:0/4160324685 >> 172.21.15.150:0/4160324685 conn(0x7fbe240797e0 msgr2=0x7fbe24079bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 -- 172.21.15.150:0/4160324685 shutdown_connections 2024-09-21T00:58:53.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.169+0000 7fbe29336640 1 -- 172.21.15.150:0/4160324685 wait complete. 2024-09-21T00:58:53.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 Processor -- start 2024-09-21T00:58:53.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 -- start start 2024-09-21T00:58:53.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe24104880 0x7fbe2419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:53.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe241052b0 0x7fbe2419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:53.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 0x7fbe2419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24113ab0 con 0x7fbe241052b0 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24113930 con 0x7fbe2410de30 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.170+0000 7fbe29336640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24113c30 con 0x7fbe24104880 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 0x7fbe2419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 0x7fbe2419fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53460/0 (socket says 172.21.15.150:53460) 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 -- 172.21.15.150:0/1527494997 learned_addr learned my addr 172.21.15.150:0/1527494997 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe227fc640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe241052b0 0x7fbe2419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe22ffd640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe24104880 0x7fbe2419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 -- 172.21.15.150:0/1527494997 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe24104880 msgr2=0x7fbe2419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe24104880 0x7fbe2419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 -- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe241052b0 msgr2=0x7fbe2419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe241052b0 0x7fbe2419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 -- 172.21.15.150:0/1527494997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe241a0310 con 0x7fbe2410de30 2024-09-21T00:58:53.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe237fe640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 0x7fbe2419fca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00cce0 tx=0x7fbe1c00a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:58:53.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c00f450 con 0x7fbe2410de30 2024-09-21T00:58:53.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe241a0600 con 0x7fbe2410de30 2024-09-21T00:58:53.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1c00faa0 con 0x7fbe2410de30 2024-09-21T00:58:53.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c018b70 con 0x7fbe2410de30 2024-09-21T00:58:53.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.171+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe24102670 con 0x7fbe2410de30 2024-09-21T00:58:53.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.173+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2406bec0 con 0x7fbe2410de30 2024-09-21T00:58:53.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.174+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fbe1c018d10 con 0x7fbe2410de30 2024-09-21T00:58:53.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.175+0000 7fbe0bfff640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbe0c0ba9a0 0x7fbe0c0bceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:58:53.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.175+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6476+0+0 (secure 0 0 0) 0x7fbe1c0e7060 con 0x7fbe2410de30 2024-09-21T00:58:53.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.175+0000 7fbe22ffd640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbe0c0ba9a0 0x7fbe0c0bceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:58:53.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.176+0000 7fbe22ffd640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbe0c0ba9a0 0x7fbe0c0bceb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe2406c5c0 tx=0x7fbe14005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:58:53.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.178+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbe1c014030 con 0x7fbe2410de30 2024-09-21T00:58:53.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:58:53.374+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbe241080e0 con 0x7fbe0c0ba9a0 2024-09-21T00:58:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: osdmap e61: 10 total, 10 up, 10 in 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='client.24584 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi039 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 4272M 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:53.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:53.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: osdmap e61: 10 total, 10 up, 10 in 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='client.24584 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi039 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 4272M 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:53.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:53.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:53.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-09-21T00:58:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: osdmap e61: 10 total, 10 up, 10 in 2024-09-21T00:58:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='client.24584 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:58:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:58:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi039 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 4272M 2024-09-21T00:58:54.109 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:54.109 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:58:54.109 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:58:54.109 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:58:54.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:54 smithi150 ceph-mon[34945]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:55.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:54 smithi160 ceph-mon[34696]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:55.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:54 smithi039 ceph-mon[30415]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:56.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:55 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/4201006946' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"} : dispatch 2024-09-21T00:58:56.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:55 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"} : dispatch 2024-09-21T00:58:56.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:55 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"}]': finished 2024-09-21T00:58:56.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:55 smithi150 ceph-mon[34945]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T00:58:56.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:55 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:58:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:55 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/4201006946' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"} : dispatch 2024-09-21T00:58:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:55 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"} : dispatch 2024-09-21T00:58:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:55 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"}]': finished 2024-09-21T00:58:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:55 smithi160 ceph-mon[34696]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T00:58:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:55 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:58:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:55 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/4201006946' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"} : dispatch 2024-09-21T00:58:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:55 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"} : dispatch 2024-09-21T00:58:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:55 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac8cd3a-cffc-466d-8214-8a3f714f5380"}]': finished 2024-09-21T00:58:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:55 smithi039 ceph-mon[30415]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T00:58:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:55 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:58:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:56 smithi150 ceph-mon[34945]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.150:0/432353090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:58:57.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:56 smithi160 ceph-mon[34696]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:57.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.150:0/432353090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:58:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:56 smithi039 ceph-mon[30415]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.150:0/432353090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:58:59.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:58:58 smithi150 ceph-mon[34945]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:59.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:58:58 smithi160 ceph-mon[34696]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:58:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:58:58 smithi039 ceph-mon[30415]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:01.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:00 smithi150 ceph-mon[34945]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:01.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:00 smithi160 ceph-mon[34696]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:01.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:00 smithi039 ceph-mon[30415]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:03.073 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:02 smithi039 ceph-mon[30415]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:03.230 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:02 smithi150 ceph-mon[34945]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:03.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:02 smithi160 ceph-mon[34696]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:05.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:05 smithi150 ceph-mon[34945]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:05.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T00:59:05.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:05.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:05 smithi160 ceph-mon[34696]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:05.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T00:59:05.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:05 smithi039 ceph-mon[30415]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T00:59:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:06 smithi150 ceph-mon[34945]: Deploying daemon osd.10 on smithi150 2024-09-21T00:59:06.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:06 smithi160 ceph-mon[34696]: Deploying daemon osd.10 on smithi150 2024-09-21T00:59:06.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:06 smithi039 ceph-mon[30415]: Deploying daemon osd.10 on smithi150 2024-09-21T00:59:07.436 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:07 smithi150 ceph-mon[34945]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:07 smithi160 ceph-mon[34696]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:07 smithi039 ceph-mon[30415]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:09.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:09 smithi150 ceph-mon[34945]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:09.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:09 smithi160 ceph-mon[34696]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:09 smithi039 ceph-mon[30415]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:11.253 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:11 smithi150 ceph-mon[34945]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:11.253 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:11.254 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:11.254 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:11 smithi160 ceph-mon[34696]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:11 smithi039 ceph-mon[30415]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:13.333 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:13 smithi150 ceph-mon[34945]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:13.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:13 smithi160 ceph-mon[34696]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:13 smithi039 ceph-mon[30415]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:14.313 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 10 on host 'smithi150' 2024-09-21T00:59:14.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.312+0000 7fbe0bfff640 1 -- 172.21.15.150:0/1527494997 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fbe241080e0 con 0x7fbe0c0ba9a0 2024-09-21T00:59:14.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.314+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbe0c0ba9a0 msgr2=0x7fbe0c0bceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:14.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.314+0000 7fbe29336640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbe0c0ba9a0 0x7fbe0c0bceb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe2406c5c0 tx=0x7fbe14005f90 comp rx=0 tx=0).stop 2024-09-21T00:59:14.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 msgr2=0x7fbe2419fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 0x7fbe2419fca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00cce0 tx=0x7fbe1c00a620 comp rx=0 tx=0).stop 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 shutdown_connections 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbe0c0ba9a0 0x7fbe0c0bceb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe2410de30 0x7fbe2419fca0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe241052b0 0x7fbe2419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 --2- 172.21.15.150:0/1527494997 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbe24104880 0x7fbe2419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 >> 172.21.15.150:0/1527494997 conn(0x7fbe240797e0 msgr2=0x7fbe241079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 shutdown_connections 2024-09-21T00:59:14.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-21T00:59:14.315+0000 7fbe29336640 1 -- 172.21.15.150:0/1527494997 wait complete. 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:14.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:15.008 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:15.008 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:15.008 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:15.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:15.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:15.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:15.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:15.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:15.039 DEBUG:teuthology.orchestra.run.smithi150:osd.10> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.10.service 2024-09-21T00:59:15.041 INFO:tasks.cephadm:Deploying osd.11 on smithi160 with /dev/vg_nvme/lv_1... 2024-09-21T00:59:15.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T00:59:15.300 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:59:15.675 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:15 smithi160 ceph-mon[34696]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:15 smithi039 ceph-mon[30415]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:16.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:15 smithi150 ceph-mon[34945]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:16.564 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 00:59:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T00:59:16.399+0000 7fe153b12640 -1 osd.10 0 log_to_monitors true 2024-09-21T00:59:16.721 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:16 smithi160 ceph-mon[34696]: from='osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T00:59:16.721 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:16 smithi160 ceph-mon[34696]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T00:59:16.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:16 smithi039 ceph-mon[30415]: from='osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T00:59:16.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:16 smithi039 ceph-mon[30415]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T00:59:17.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:16 smithi150 ceph-mon[34945]: from='osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T00:59:17.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:16 smithi150 ceph-mon[34945]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T00:59:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T00:59:17.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_1 2024-09-21T00:59:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:17 smithi039 ceph-mon[30415]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:17 smithi039 ceph-mon[30415]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T00:59:17.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:17 smithi039 ceph-mon[30415]: from='osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:59:17.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:17 smithi039 ceph-mon[30415]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T00:59:17.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:17.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:17 smithi039 ceph-mon[30415]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:59:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T00:59:17.623+0000 7fe14fa94640 -1 osd.10 0 waiting for initial osdmap 2024-09-21T00:59:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T00:59:17.635+0000 7fe14b892640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:59:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-mon[34945]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-mon[34945]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T00:59:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-mon[34945]: from='osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:59:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-mon[34945]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T00:59:18.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:18.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:17 smithi150 ceph-mon[34945]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:59:18.087 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:59:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:17 smithi160 ceph-mon[34696]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:17 smithi160 ceph-mon[34696]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T00:59:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:17 smithi160 ceph-mon[34696]: from='osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:59:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:17 smithi160 ceph-mon[34696]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T00:59:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:17 smithi160 ceph-mon[34696]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]} : dispatch 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 -- 172.21.15.160:0/299911909 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc102ae0 msgr2=0x7fb4bc102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/299911909 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc102ae0 0x7fb4bc102ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009900 tx=0x7fb4ac02a410 comp rx=0 tx=0).stop 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 -- 172.21.15.160:0/299911909 shutdown_connections 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/299911909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4bc108d70 0x7fb4bc10f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/299911909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108420 0x7fb4bc108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/299911909 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc102ae0 0x7fb4bc102ef0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 -- 172.21.15.160:0/299911909 >> 172.21.15.160:0/299911909 conn(0x7fb4bc079770 msgr2=0x7fb4bc079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 -- 172.21.15.160:0/299911909 shutdown_connections 2024-09-21T00:59:18.578 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.577+0000 7fb4c1f71640 1 -- 172.21.15.160:0/299911909 wait complete. 2024-09-21T00:59:18.580 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 Processor -- start 2024-09-21T00:59:18.580 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 -- start start 2024-09-21T00:59:18.580 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4bc102ae0 0x7fb4bc19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:18.580 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 0x7fb4bc19b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:18.580 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108d70 0x7fb4bc19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc112a30 con 0x7fb4bc108d70 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc1128b0 con 0x7fb4bc102ae0 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4c1f71640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc112bb0 con 0x7fb4bc108420 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4baffd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 0x7fb4bc19b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4baffd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 0x7fb4bc19b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:36894/0 (socket says 172.21.15.160:36894) 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4baffd640 1 -- 172.21.15.160:0/2402737509 learned_addr learned my addr 172.21.15.160:0/2402737509 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.578+0000 7fb4bb7fe640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4bc102ae0 0x7fb4bc19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4bbfff640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108d70 0x7fb4bc19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4baffd640 1 -- 172.21.15.160:0/2402737509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4bc102ae0 msgr2=0x7fb4bc19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4baffd640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4bc102ae0 0x7fb4bc19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4baffd640 1 -- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108d70 msgr2=0x7fb4bc19be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4baffd640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108d70 0x7fb4bc19be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:18.581 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4baffd640 1 -- 172.21.15.160:0/2402737509 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4bc1a6610 con 0x7fb4bc108420 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4bbfff640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108d70 0x7fb4bc19be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4baffd640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 0x7fb4bc19b8c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000d740 tx=0x7fb4b000dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0017070 con 0x7fb4bc108420 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4b000de50 con 0x7fb4bc108420 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b000f620 con 0x7fb4bc108420 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4bc1a6900 con 0x7fb4bc108420 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.579+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4bc1a6e90 con 0x7fb4bc108420 2024-09-21T00:59:18.582 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.580+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4bc103d60 con 0x7fb4bc108420 2024-09-21T00:59:18.583 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.582+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb4b001a070 con 0x7fb4bc108420 2024-09-21T00:59:18.583 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.583+0000 7fb4b8ff9640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb4a40b6610 0x7fb4a40b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:18.583 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.583+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6629+0+0 (secure 0 0 0) 0x7fb4b00e6610 con 0x7fb4bc108420 2024-09-21T00:59:18.583 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.583+0000 7fb4bb7fe640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb4a40b6610 0x7fb4a40b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:18.585 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.584+0000 7fb4bb7fe640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb4a40b6610 0x7fb4a40b8b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac000c00 tx=0x7fb4ac031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:18.586 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.586+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb4b00ad5b0 con 0x7fb4bc108420 2024-09-21T00:59:18.782 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:18.782+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb4bc063800 con 0x7fb4a40b6610 2024-09-21T00:59:18.783 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:59:18.783 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: osdmap e64: 11 total, 10 up, 11 in 2024-09-21T00:59:18.783 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:18.783 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:18.785 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:18.785 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:18 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:59:18.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: osdmap e64: 11 total, 10 up, 11 in 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:18.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:18 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:19.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-09-21T00:59:19.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: osdmap e64: 11 total, 10 up, 11 in 2024-09-21T00:59:19.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:19.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:19.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:19.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:19.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:18 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:59:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:59:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi150 2024-09-21T00:59:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi150 to 4272M 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] boot 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:19.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:19 smithi160 ceph-mon[34696]: osdmap e66: 11 total, 11 up, 11 in 2024-09-21T00:59:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:59:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi150 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi150 to 4272M 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] boot 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:19.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:19 smithi039 ceph-mon[30415]: osdmap e66: 11 total, 11 up, 11 in 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi150 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi150 to 4272M 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: osd.10 [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] boot 2024-09-21T00:59:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T00:59:20.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T00:59:20.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T00:59:20.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T00:59:20.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:20.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:19 smithi150 ceph-mon[34945]: osdmap e66: 11 total, 11 up, 11 in 2024-09-21T00:59:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:20 smithi160 ceph-mon[34696]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:20 smithi160 ceph-mon[34696]: osdmap e67: 11 total, 11 up, 11 in 2024-09-21T00:59:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:20 smithi039 ceph-mon[30415]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:20 smithi039 ceph-mon[30415]: osdmap e67: 11 total, 11 up, 11 in 2024-09-21T00:59:21.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:20 smithi150 ceph-mon[34945]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:21.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:20 smithi150 ceph-mon[34945]: osdmap e67: 11 total, 11 up, 11 in 2024-09-21T00:59:21.837 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:21 smithi160 ceph-mon[34696]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:21.837 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:21 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/164101989' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"} : dispatch 2024-09-21T00:59:21.837 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:21 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"} : dispatch 2024-09-21T00:59:21.837 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:21 smithi160 ceph-mon[34696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"}]': finished 2024-09-21T00:59:21.837 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:21 smithi160 ceph-mon[34696]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T00:59:21.837 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:21 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:21 smithi039 ceph-mon[30415]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:21 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/164101989' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"} : dispatch 2024-09-21T00:59:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:21 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"} : dispatch 2024-09-21T00:59:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:21 smithi039 ceph-mon[30415]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"}]': finished 2024-09-21T00:59:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:21 smithi039 ceph-mon[30415]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T00:59:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:21 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:21 smithi150 ceph-mon[34945]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:21 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/164101989' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"} : dispatch 2024-09-21T00:59:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:21 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"} : dispatch 2024-09-21T00:59:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:21 smithi150 ceph-mon[34945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0574abe-bba5-4251-bfd6-2610607a2400"}]': finished 2024-09-21T00:59:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:21 smithi150 ceph-mon[34945]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T00:59:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:21 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:22.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:22 smithi039 ceph-mon[30415]: from='client.? 172.21.15.160:0/370883469' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:59:23.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:22 smithi150 ceph-mon[34945]: from='client.? 172.21.15.160:0/370883469' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:59:23.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:22 smithi160 ceph-mon[34696]: from='client.? 172.21.15.160:0/370883469' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T00:59:23.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:23 smithi039 ceph-mon[30415]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:23 smithi150 ceph-mon[34945]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:23 smithi160 ceph-mon[34696]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:25 smithi039 ceph-mon[30415]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:25.916 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:25 smithi160 ceph-mon[34696]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:25 smithi150 ceph-mon[34945]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:26.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:26 smithi160 ceph-mon[34696]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T00:59:27.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:26 smithi150 ceph-mon[34945]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T00:59:27.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:26 smithi039 ceph-mon[30415]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T00:59:29.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:28 smithi150 ceph-mon[34945]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T00:59:29.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:28 smithi160 ceph-mon[34696]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T00:59:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:28 smithi039 ceph-mon[30415]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T00:59:30.194 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T00:59:30.194 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:29 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:30.194 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:29 smithi160 ceph-mon[34696]: Deploying daemon osd.11 on smithi160 2024-09-21T00:59:30.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T00:59:30.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:29 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:30.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:29 smithi150 ceph-mon[34945]: Deploying daemon osd.11 on smithi160 2024-09-21T00:59:30.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T00:59:30.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:29 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:30.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:29 smithi039 ceph-mon[30415]: Deploying daemon osd.11 on smithi160 2024-09-21T00:59:31.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:30 smithi160 ceph-mon[34696]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T00:59:31.304 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:30 smithi039 ceph-mon[30415]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T00:59:31.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:30 smithi150 ceph-mon[34945]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T00:59:33.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:32 smithi160 ceph-mon[34696]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-21T00:59:33.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:32 smithi150 ceph-mon[34945]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-21T00:59:33.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:32 smithi039 ceph-mon[30415]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-21T00:59:35.099 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:34 smithi160 ceph-mon[34696]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:59:35.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:34 smithi150 ceph-mon[34945]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:59:35.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:34 smithi039 ceph-mon[30415]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:59:36.081 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:36.082 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:36.082 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:36.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:36.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:36.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:37.601 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:37 smithi160 ceph-mon[34696]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:59:37.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:37 smithi039 ceph-mon[30415]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:59:37.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:37 smithi150 ceph-mon[34945]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T00:59:38.668 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.668+0000 7fb4b8ff9640 1 -- 172.21.15.160:0/2402737509 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb4bc063800 con 0x7fb4a40b6610 2024-09-21T00:59:38.669 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 11 on host 'smithi160' 2024-09-21T00:59:38.671 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb4a40b6610 msgr2=0x7fb4a40b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:38.671 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb4a40b6610 0x7fb4a40b8b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac000c00 tx=0x7fb4ac031400 comp rx=0 tx=0).stop 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 msgr2=0x7fb4bc19b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 0x7fb4bc19b8c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000d740 tx=0x7fb4b000dc00 comp rx=0 tx=0).stop 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 shutdown_connections 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4bc108d70 0x7fb4bc19be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4bc108420 0x7fb4bc19b8c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb4a40b6610 0x7fb4a40b8b20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 --2- 172.21.15.160:0/2402737509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4bc102ae0 0x7fb4bc19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:38.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.670+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 >> 172.21.15.160:0/2402737509 conn(0x7fb4bc079770 msgr2=0x7fb4bc106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:38.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.671+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 shutdown_connections 2024-09-21T00:59:38.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:38.671+0000 7fb4c1f71640 1 -- 172.21.15.160:0/2402737509 wait complete. 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.327 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.355 DEBUG:teuthology.orchestra.run.smithi160:osd.11> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.11.service 2024-09-21T00:59:39.357 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-21T00:59:39.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T00:59:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:39.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:39.747 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:40.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 -- 172.21.15.39:0/3732772081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010b6c0 msgr2=0x7fb7b010bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:40.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/3732772081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010b6c0 0x7fb7b010bad0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb798009900 tx=0x7fb79802a410 comp rx=0 tx=0).stop 2024-09-21T00:59:40.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 -- 172.21.15.39:0/3732772081 shutdown_connections 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/3732772081 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b010c010 0x7fb7b0112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/3732772081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010b6c0 0x7fb7b010bad0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/3732772081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b0105d80 0x7fb7b0106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 -- 172.21.15.39:0/3732772081 >> 172.21.15.39:0/3732772081 conn(0x7fb7b0101810 msgr2=0x7fb7b0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 -- 172.21.15.39:0/3732772081 shutdown_connections 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.328+0000 7fb7b4a00640 1 -- 172.21.15.39:0/3732772081 wait complete. 2024-09-21T00:59:40.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.329+0000 7fb7b4a00640 1 Processor -- start 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.329+0000 7fb7b4a00640 1 -- start start 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7b4a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 0x7fb7b019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7b4a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b010b6c0 0x7fb7b019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7b4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 0x7fb7b01a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7b4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0115d20 con 0x7fb7b010c010 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7b4a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0115ba0 con 0x7fb7b010b6c0 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7b4a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0115ea0 con 0x7fb7b0105d80 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 0x7fb7b01a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7ade00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 0x7fb7b019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:40.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7a7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b010b6c0 0x7fb7b019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7ade00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 0x7fb7b019c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:35774/0 (socket says 172.21.15.39:35774) 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 0x7fb7b01a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35560/0 (socket says 172.21.15.39:35560) 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.330+0000 7fb7ade00640 1 -- 172.21.15.39:0/1932278068 learned_addr learned my addr 172.21.15.39:0/1932278068 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a7e00640 1 -- 172.21.15.39:0/1932278068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 msgr2=0x7fb7b019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a7e00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 0x7fb7b019c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a7e00640 1 -- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 msgr2=0x7fb7b01a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a7e00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 0x7fb7b01a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a7e00640 1 -- 172.21.15.39:0/1932278068 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b01a2620 con 0x7fb7b010b6c0 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7ae800640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 0x7fb7b01a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7ade00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 0x7fb7b019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a7e00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b010b6c0 0x7fb7b019cc50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb798009230 tx=0x7fb798003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.331+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79803f070 con 0x7fb7b010b6c0 2024-09-21T00:59:40.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.332+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb798031b50 con 0x7fb7b010b6c0 2024-09-21T00:59:40.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.332+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b01a28b0 con 0x7fb7b010b6c0 2024-09-21T00:59:40.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.332+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798035030 con 0x7fb7b010b6c0 2024-09-21T00:59:40.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.332+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b01a9070 con 0x7fb7b010b6c0 2024-09-21T00:59:40.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.333+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b0107000 con 0x7fb7b010b6c0 2024-09-21T00:59:40.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.335+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb798042070 con 0x7fb7b010b6c0 2024-09-21T00:59:40.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.337+0000 7fb7a6a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7940b6560 0x7fb7940b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:40.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.338+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 7024+0+0 (secure 0 0 0) 0x7fb798107880 con 0x7fb7b010b6c0 2024-09-21T00:59:40.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.338+0000 7fb7ade00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7940b6560 0x7fb7940b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:40.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.338+0000 7fb7ade00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7940b6560 0x7fb7940b8a70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0011550 tx=0x7fb7a0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:40.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.339+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb7980d2980 con 0x7fb7b010b6c0 2024-09-21T00:59:40.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.537+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb7b010de00 con 0x7fb7b010b6c0 2024-09-21T00:59:40.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.538+0000 7fb7a6a00640 1 -- 172.21.15.39:0/1932278068 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v68) ==== 74+0+133 (secure 0 0 0) 0x7fb7980d78c0 con 0x7fb7b010b6c0 2024-09-21T00:59:40.539 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.540+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7940b6560 msgr2=0x7fb7940b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.540+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7940b6560 0x7fb7940b8a70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0011550 tx=0x7fb7a0005f90 comp rx=0 tx=0).stop 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.540+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b010b6c0 msgr2=0x7fb7b019cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.540+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b010b6c0 0x7fb7b019cc50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb798009230 tx=0x7fb798003b90 comp rx=0 tx=0).stop 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 shutdown_connections 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7b010c010 0x7fb7b01a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb7940b6560 0x7fb7940b8a70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b010b6c0 0x7fb7b019cc50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 --2- 172.21.15.39:0/1932278068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb7b0105d80 0x7fb7b019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 >> 172.21.15.39:0/1932278068 conn(0x7fb7b0101810 msgr2=0x7fb7b0108cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:40.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 shutdown_connections 2024-09-21T00:59:40.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:40.541+0000 7fb7b4a00640 1 -- 172.21.15.39:0/1932278068 wait complete. 2024-09-21T00:59:41.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:40 smithi160 ceph-mon[34696]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:41.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:40 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1932278068' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:41.208 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:40 smithi039 ceph-mon[30415]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:41.208 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:40 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1932278068' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:40 smithi150 ceph-mon[34945]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:40 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1932278068' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:41.516 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 00:59:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T00:59:41.183+0000 7fe95bcc6640 -1 osd.11 0 log_to_monitors true 2024-09-21T00:59:41.685 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":68,"num_osds":12,"num_up_osds":11,"osd_up_since":1726880358,"num_in_osds":12,"osd_in_since":1726880361,"num_remapped_pgs":0} 2024-09-21T00:59:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:41 smithi160 ceph-mon[34696]: from='osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T00:59:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:41 smithi160 ceph-mon[34696]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T00:59:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:41 smithi150 ceph-mon[34945]: from='osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T00:59:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:41 smithi150 ceph-mon[34945]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T00:59:42.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:41 smithi039 ceph-mon[30415]: from='osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T00:59:42.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:41 smithi039 ceph-mon[30415]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T00:59:42.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T00:59:43.067 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:43.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:43.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T00:59:43.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: osdmap e69: 12 total, 11 up, 12 in 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: Detected new or changed devices on smithi160 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:43.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:43.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T00:59:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:59:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: osdmap e69: 12 total, 11 up, 12 in 2024-09-21T00:59:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: Detected new or changed devices on smithi160 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:43.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T00:59:42.978+0000 7fe957c48640 -1 osd.11 0 waiting for initial osdmap 2024-09-21T00:59:43.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T00:59:42.997+0000 7fe953a46640 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T00:59:43.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:43.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T00:59:43.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:59:43.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: osdmap e69: 12 total, 11 up, 12 in 2024-09-21T00:59:43.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:43.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: Detected new or changed devices on smithi160 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:43.358 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:43.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.703+0000 7ff44f200640 1 -- 172.21.15.39:0/810251151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45010ff10 msgr2=0x7ff450112350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:43.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.703+0000 7ff44f200640 1 --2- 172.21.15.39:0/810251151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45010ff10 0x7ff450112350 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff448009900 tx=0x7ff44802a410 comp rx=0 tx=0).stop 2024-09-21T00:59:43.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 -- 172.21.15.39:0/810251151 shutdown_connections 2024-09-21T00:59:43.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 --2- 172.21.15.39:0/810251151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45010ff10 0x7ff450112350 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 --2- 172.21.15.39:0/810251151 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff450076100 0x7ff45010f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 --2- 172.21.15.39:0/810251151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4500757b0 0x7ff450075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 -- 172.21.15.39:0/810251151 >> 172.21.15.39:0/810251151 conn(0x7ff4500ff440 msgr2=0x7ff450101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:43.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 -- 172.21.15.39:0/810251151 shutdown_connections 2024-09-21T00:59:43.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.704+0000 7ff44f200640 1 -- 172.21.15.39:0/810251151 wait complete. 2024-09-21T00:59:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 Processor -- start 2024-09-21T00:59:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 -- start start 2024-09-21T00:59:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 0x7ff45019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff450076100 0x7ff45019c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff45010ff10 0x7ff4501a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff450115ac0 con 0x7ff4500757b0 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff450115940 con 0x7ff450076100 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.705+0000 7ff44f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff450115c40 con 0x7ff45010ff10 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 0x7ff45019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 0x7ff45019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47554/0 (socket says 172.21.15.39:47554) 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 -- 172.21.15.39:0/1071682128 learned_addr learned my addr 172.21.15.39:0/1071682128 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44d400640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff450076100 0x7ff45019c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44e800640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff45010ff10 0x7ff4501a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 -- 172.21.15.39:0/1071682128 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff45010ff10 msgr2=0x7ff4501a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff45010ff10 0x7ff4501a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 -- 172.21.15.39:0/1071682128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff450076100 msgr2=0x7ff45019c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff450076100 0x7ff45019c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44de00640 1 -- 172.21.15.39:0/1071682128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4501a23c0 con 0x7ff4500757b0 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44d400640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff450076100 0x7ff45019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:59:43.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.706+0000 7ff44e800640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff45010ff10 0x7ff4501a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.707+0000 7ff44de00640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 0x7ff45019c450 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff44000ebc0 tx=0x7ff44000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.707+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440010630 con 0x7ff4500757b0 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.707+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff440010c80 con 0x7ff4500757b0 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.707+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440018b60 con 0x7ff4500757b0 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.707+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4501a26b0 con 0x7ff4500757b0 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.707+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4501a8e10 con 0x7ff4500757b0 2024-09-21T00:59:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.708+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff450075bc0 con 0x7ff4500757b0 2024-09-21T00:59:43.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.711+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff440018d00 con 0x7ff4500757b0 2024-09-21T00:59:43.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.712+0000 7ff446a00640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4340b65e0 0x7ff4340b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:43.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.712+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7062+0+0 (secure 0 0 0) 0x7ff4400e7920 con 0x7ff4500757b0 2024-09-21T00:59:43.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.712+0000 7ff44d400640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4340b65e0 0x7ff4340b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:43.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.713+0000 7ff44d400640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4340b65e0 0x7ff4340b8af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff450112350 tx=0x7ff438005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:43.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.714+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff4400b29d0 con 0x7ff4500757b0 2024-09-21T00:59:43.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.911+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ff4501a91c0 con 0x7ff4500757b0 2024-09-21T00:59:43.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.912+0000 7ff446a00640 1 -- 172.21.15.39:0/1071682128 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7ff4400b7910 con 0x7ff4500757b0 2024-09-21T00:59:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4340b65e0 msgr2=0x7ff4340b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4340b65e0 0x7ff4340b8af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff450112350 tx=0x7ff438005f90 comp rx=0 tx=0).stop 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 msgr2=0x7ff45019c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 0x7ff45019c450 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff44000ebc0 tx=0x7ff44000c6a0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 shutdown_connections 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff45010ff10 0x7ff4501a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4340b65e0 0x7ff4340b8af0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff450076100 0x7ff45019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 --2- 172.21.15.39:0/1071682128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500757b0 0x7ff45019c450 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.914+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 >> 172.21.15.39:0/1071682128 conn(0x7ff4500ff440 msgr2=0x7ff450109ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:43.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.915+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 shutdown_connections 2024-09-21T00:59:43.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:43.915+0000 7ff44f200640 1 -- 172.21.15.39:0/1071682128 wait complete. 2024-09-21T00:59:44.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:43 smithi039 ceph-mon[30415]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:59:44.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:43 smithi039 ceph-mon[30415]: osdmap e70: 12 total, 11 up, 12 in 2024-09-21T00:59:44.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:44.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:43 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1071682128' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:44.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:43 smithi150 ceph-mon[34945]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:59:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:43 smithi150 ceph-mon[34945]: osdmap e70: 12 total, 11 up, 12 in 2024-09-21T00:59:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:43 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1071682128' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:43 smithi160 ceph-mon[34696]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T00:59:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:43 smithi160 ceph-mon[34696]: osdmap e70: 12 total, 11 up, 12 in 2024-09-21T00:59:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:43 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1071682128' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:44.851 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1726880358,"num_in_osds":12,"osd_in_since":1726880361,"num_remapped_pgs":0} 2024-09-21T00:59:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:44 smithi039 ceph-mon[30415]: purged_snaps scrub starts 2024-09-21T00:59:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:44 smithi039 ceph-mon[30415]: purged_snaps scrub ok 2024-09-21T00:59:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:44 smithi039 ceph-mon[30415]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:44 smithi039 ceph-mon[30415]: osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] boot 2024-09-21T00:59:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:44 smithi039 ceph-mon[30415]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T00:59:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:44 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:44 smithi150 ceph-mon[34945]: purged_snaps scrub starts 2024-09-21T00:59:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:44 smithi150 ceph-mon[34945]: purged_snaps scrub ok 2024-09-21T00:59:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:44 smithi150 ceph-mon[34945]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:44 smithi150 ceph-mon[34945]: osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] boot 2024-09-21T00:59:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:44 smithi150 ceph-mon[34945]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T00:59:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:44 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:45.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:44 smithi160 ceph-mon[34696]: purged_snaps scrub starts 2024-09-21T00:59:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:44 smithi160 ceph-mon[34696]: purged_snaps scrub ok 2024-09-21T00:59:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:44 smithi160 ceph-mon[34696]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-09-21T00:59:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:44 smithi160 ceph-mon[34696]: osd.11 [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] boot 2024-09-21T00:59:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:44 smithi160 ceph-mon[34696]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T00:59:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:44 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T00:59:45.852 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T00:59:46.238 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:46 smithi150 ceph-mon[34945]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T00:59:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:46 smithi160 ceph-mon[34696]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T00:59:46.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:46 smithi039 ceph-mon[30415]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 -- 172.21.15.39:0/3891111969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0090103990 msgr2=0x7f0090103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 --2- 172.21.15.39:0/3891111969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0090103990 0x7f0090103da0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0080009900 tx=0x7f008002a410 comp rx=0 tx=0).stop 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 -- 172.21.15.39:0/3891111969 shutdown_connections 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 --2- 172.21.15.39:0/3891111969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0090109c20 0x7f0090110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 --2- 172.21.15.39:0/3891111969 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00901092d0 0x7f00901096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 --2- 172.21.15.39:0/3891111969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0090103990 0x7f0090103da0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 -- 172.21.15.39:0/3891111969 >> 172.21.15.39:0/3891111969 conn(0x7f00900ff440 msgr2=0x7f0090101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 -- 172.21.15.39:0/3891111969 shutdown_connections 2024-09-21T00:59:46.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.887+0000 7f0094e00640 1 -- 172.21.15.39:0/3891111969 wait complete. 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.888+0000 7f0094e00640 1 Processor -- start 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.888+0000 7f0094e00640 1 -- start start 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0094e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0090103990 0x7f009019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0094e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 0x7f009019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0094e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0090109c20 0x7f00901a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0094e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f00901138e0 con 0x7f00901092d0 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0094e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0090113760 con 0x7f0090103990 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0094e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0090113a60 con 0x7f0090109c20 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0086a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 0x7f009019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:46.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0086a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 0x7f009019ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47568/0 (socket says 172.21.15.39:47568) 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f0086a00640 1 -- 172.21.15.39:0/2026172283 learned_addr learned my addr 172.21.15.39:0/2026172283 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f008f400640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0090103990 0x7f009019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.889+0000 7f008fe00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0090109c20 0x7f00901a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0086a00640 1 -- 172.21.15.39:0/2026172283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0090109c20 msgr2=0x7f00901a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0086a00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0090109c20 0x7f00901a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0086a00640 1 -- 172.21.15.39:0/2026172283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0090103990 msgr2=0x7f009019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0086a00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0090103990 0x7f009019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:46.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0086a00640 1 -- 172.21.15.39:0/2026172283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00901a2460 con 0x7f00901092d0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0086a00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 0x7f009019ca90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f007800d950 tx=0x7f007800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0078017070 con 0x7f00901092d0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0078009c30 con 0x7f00901092d0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00901a2750 con 0x7f00901092d0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.890+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00780047a0 con 0x7f00901092d0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.891+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00901a8eb0 con 0x7f00901092d0 2024-09-21T00:59:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.892+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0090103da0 con 0x7f00901092d0 2024-09-21T00:59:46.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.895+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f0078004020 con 0x7f00901092d0 2024-09-21T00:59:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.898+0000 7f008cc00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00740b6560 0x7f00740b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.898+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f00780e7080 con 0x7f00901092d0 2024-09-21T00:59:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.898+0000 7f008f400640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00740b6560 0x7f00740b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:46.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.899+0000 7f008f400640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00740b6560 0x7f00740b8a70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0080031440 tx=0x7f0080009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:46.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:46.899+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0078020020 con 0x7f00901092d0 2024-09-21T00:59:47.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.100+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f00901a9110 con 0x7f00901092d0 2024-09-21T00:59:47.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.101+0000 7f008cc00640 1 -- 172.21.15.39:0/2026172283 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f00780adc40 con 0x7f00901092d0 2024-09-21T00:59:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00740b6560 msgr2=0x7f00740b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00740b6560 0x7f00740b8a70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0080031440 tx=0x7f0080009560 comp rx=0 tx=0).stop 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 msgr2=0x7f009019ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 0x7f009019ca90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f007800d950 tx=0x7f007800de10 comp rx=0 tx=0).stop 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 shutdown_connections 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0090109c20 0x7f00901a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00901092d0 0x7f009019ca90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00740b6560 0x7f00740b8a70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 --2- 172.21.15.39:0/2026172283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0090103990 0x7f009019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:47.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 >> 172.21.15.39:0/2026172283 conn(0x7f00900ff440 msgr2=0x7f0090108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:47.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.103+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 shutdown_connections 2024-09-21T00:59:47.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:47.104+0000 7f0094e00640 1 -- 172.21.15.39:0/2026172283 wait complete. 2024-09-21T00:59:47.276 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:47 smithi160 ceph-mon[34696]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:47.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:47 smithi150 ceph-mon[34945]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:47.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:47 smithi039 ceph-mon[30415]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:48.048 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1726880383,"num_in_osds":12,"osd_in_since":1726880361,"num_remapped_pgs":0} 2024-09-21T00:59:48.049 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T00:59:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:48 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2026172283' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:48 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2026172283' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:48.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:48 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2026172283' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T00:59:48.424 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 -- 172.21.15.39:0/3488230042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a20105d80 msgr2=0x7f4a20106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3488230042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a20105d80 0x7f4a20106190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009900 tx=0x7f4a1002a410 comp rx=0 tx=0).stop 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 -- 172.21.15.39:0/3488230042 shutdown_connections 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3488230042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2010c010 0x7f4a20112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3488230042 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a2010b6c0 0x7f4a2010bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3488230042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a20105d80 0x7f4a20106190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 -- 172.21.15.39:0/3488230042 >> 172.21.15.39:0/3488230042 conn(0x7f4a20101810 msgr2=0x7f4a20103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:49.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.042+0000 7f4a26a00640 1 -- 172.21.15.39:0/3488230042 shutdown_connections 2024-09-21T00:59:49.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.043+0000 7f4a26a00640 1 -- 172.21.15.39:0/3488230042 wait complete. 2024-09-21T00:59:49.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 Processor -- start 2024-09-21T00:59:49.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 -- start start 2024-09-21T00:59:49.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a20105d80 0x7f4a2019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:49.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2010b6c0 0x7f4a2019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:49.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2010c010 0x7f4a201a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:49.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4a20115d20 con 0x7f4a2010b6c0 2024-09-21T00:59:49.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a20115ba0 con 0x7f4a2010c010 2024-09-21T00:59:49.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a26a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4a20115ea0 con 0x7f4a20105d80 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2010b6c0 0x7f4a2019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.044+0000 7f4a1fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a20105d80 0x7f4a2019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2010b6c0 0x7f4a2019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47592/0 (socket says 172.21.15.39:47592) 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1f400640 1 -- 172.21.15.39:0/3240527747 learned_addr learned my addr 172.21.15.39:0/3240527747 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a24a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2010c010 0x7f4a201a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1fe00640 1 -- 172.21.15.39:0/3240527747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2010c010 msgr2=0x7f4a201a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:49.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1fe00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2010c010 0x7f4a201a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1fe00640 1 -- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2010b6c0 msgr2=0x7f4a2019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1fe00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2010b6c0 0x7f4a2019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1fe00640 1 -- 172.21.15.39:0/3240527747 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a201a2660 con 0x7f4a20105d80 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.045+0000 7f4a1fe00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a20105d80 0x7f4a2019c6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a1002aa00 tx=0x7f4a10003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.046+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1003f070 con 0x7f4a20105d80 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.046+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a10031b50 con 0x7f4a20105d80 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.046+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a201a28f0 con 0x7f4a20105d80 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.046+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10035030 con 0x7f4a20105d80 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.046+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a201a90b0 con 0x7f4a20105d80 2024-09-21T00:59:49.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.047+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a20107000 con 0x7f4a20105d80 2024-09-21T00:59:49.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.049+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f4a10042070 con 0x7f4a20105d80 2024-09-21T00:59:49.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.050+0000 7f4a1cc00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4a040b6560 0x7f4a040b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:49.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.051+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f4a10107570 con 0x7f4a20105d80 2024-09-21T00:59:49.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.051+0000 7f4a1f400640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4a040b6560 0x7f4a040b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:49.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.051+0000 7f4a1f400640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4a040b6560 0x7f4a040b8a70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a2010b190 tx=0x7f4a0c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:49.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.055+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4a10031d00 con 0x7f4a20105d80 2024-09-21T00:59:49.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:49 smithi039 ceph-mon[30415]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:49.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.254+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a2010dde0 con 0x7f4a20105d80 2024-09-21T00:59:49.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.256+0000 7f4a1cc00640 1 -- 172.21.15.39:0/3240527747 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18818 (secure 0 0 0) 0x7f4a100d2510 con 0x7f4a20105d80 2024-09-21T00:59:49.257 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:59:49.257 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T00:59:44.992796+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T00:59:49.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4a040b6560 msgr2=0x7f4a040b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:49.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4a040b6560 0x7f4a040b8a70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a2010b190 tx=0x7f4a0c005f90 comp rx=0 tx=0).stop 2024-09-21T00:59:49.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a20105d80 msgr2=0x7f4a2019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a20105d80 0x7f4a2019c6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a1002aa00 tx=0x7f4a10003b90 comp rx=0 tx=0).stop 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 shutdown_connections 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4a040b6560 0x7f4a040b8a70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2010c010 0x7f4a201a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2010b6c0 0x7f4a2019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 --2- 172.21.15.39:0/3240527747 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a20105d80 0x7f4a2019c6f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 >> 172.21.15.39:0/3240527747 conn(0x7f4a20101810 msgr2=0x7f4a20103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 shutdown_connections 2024-09-21T00:59:49.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:49.258+0000 7f4a26a00640 1 -- 172.21.15.39:0/3240527747 wait complete. 2024-09-21T00:59:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:49 smithi150 ceph-mon[34945]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:49.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:49 smithi160 ceph-mon[34696]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:50.076 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T00:55:22.265381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T00:59:50.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T00:59:50.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:50 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3240527747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T00:59:50.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:50 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3240527747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T00:59:50.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:50 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3240527747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T00:59:50.448 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 -- 172.21.15.39:0/2070685588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410c010 msgr2=0x7fce74112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 --2- 172.21.15.39:0/2070685588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410c010 0x7fce74112840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009ca0 tx=0x7fce6c02a8b0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 -- 172.21.15.39:0/2070685588 shutdown_connections 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 --2- 172.21.15.39:0/2070685588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410c010 0x7fce74112840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 --2- 172.21.15.39:0/2070685588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce7410b6c0 0x7fce7410bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 --2- 172.21.15.39:0/2070685588 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce74105d80 0x7fce74106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 -- 172.21.15.39:0/2070685588 >> 172.21.15.39:0/2070685588 conn(0x7fce741017d0 msgr2=0x7fce74103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 -- 172.21.15.39:0/2070685588 shutdown_connections 2024-09-21T00:59:51.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.073+0000 7fce7b800640 1 -- 172.21.15.39:0/2070685588 wait complete. 2024-09-21T00:59:51.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.074+0000 7fce7b800640 1 Processor -- start 2024-09-21T00:59:51.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 -- start start 2024-09-21T00:59:51.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce74105d80 0x7fce7419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:51.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410b6c0 0x7fce7419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:51.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce7410c010 0x7fce741a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fce74115d20 con 0x7fce74105d80 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fce74115ba0 con 0x7fce7410b6c0 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce7b800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fce74115ea0 con 0x7fce7410c010 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce78e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce74105d80 0x7fce7419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce6be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410b6c0 0x7fce7419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce78e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce74105d80 0x7fce7419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47606/0 (socket says 172.21.15.39:47606) 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.075+0000 7fce78e00640 1 -- 172.21.15.39:0/1280146300 learned_addr learned my addr 172.21.15.39:0/1280146300 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce79800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce7410c010 0x7fce741a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce6be00640 1 -- 172.21.15.39:0/1280146300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce7410c010 msgr2=0x7fce741a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce6be00640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce7410c010 0x7fce741a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce6be00640 1 -- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce74105d80 msgr2=0x7fce7419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce6be00640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce74105d80 0x7fce7419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce6be00640 1 -- 172.21.15.39:0/1280146300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce741a2600 con 0x7fce7410b6c0 2024-09-21T00:59:51.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce6be00640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410b6c0 0x7fce7419cc30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce7406cac0 tx=0x7fce5c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:51.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.076+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c00cd00 con 0x7fce7410b6c0 2024-09-21T00:59:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.077+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce741a28f0 con 0x7fce7410b6c0 2024-09-21T00:59:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.077+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce5c0035f0 con 0x7fce7410b6c0 2024-09-21T00:59:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.077+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c010690 con 0x7fce7410b6c0 2024-09-21T00:59:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.077+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce741a9050 con 0x7fce7410b6c0 2024-09-21T00:59:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.078+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce74106190 con 0x7fce7410b6c0 2024-09-21T00:59:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.080+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fce5c0037a0 con 0x7fce7410b6c0 2024-09-21T00:59:51.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.082+0000 7fce72000640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fce580b6560 0x7fce580b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.083+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fce5c0e7420 con 0x7fce7410b6c0 2024-09-21T00:59:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.083+0000 7fce78e00640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fce580b6560 0x7fce580b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.083+0000 7fce78e00640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fce580b6560 0x7fce580b8a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce64009cd0 tx=0x7fce64005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.083+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fce5c056710 con 0x7fce7410b6c0 2024-09-21T00:59:51.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:51 smithi039 ceph-mon[30415]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:51.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.284+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fce7402e640 con 0x7fce7410b6c0 2024-09-21T00:59:51.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.284+0000 7fce72000640 1 -- 172.21.15.39:0/1280146300 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7fce5c0adfe0 con 0x7fce7410b6c0 2024-09-21T00:59:51.285 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-09-21T00:59:51.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fce580b6560 msgr2=0x7fce580b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fce580b6560 0x7fce580b8a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce64009cd0 tx=0x7fce64005f90 comp rx=0 tx=0).stop 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410b6c0 msgr2=0x7fce7419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410b6c0 0x7fce7419cc30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce7406cac0 tx=0x7fce5c00ee50 comp rx=0 tx=0).stop 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 shutdown_connections 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce7410c010 0x7fce741a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fce580b6560 0x7fce580b8a70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7410b6c0 0x7fce7419cc30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 --2- 172.21.15.39:0/1280146300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce74105d80 0x7fce7419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 >> 172.21.15.39:0/1280146300 conn(0x7fce741017d0 msgr2=0x7fce74103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 shutdown_connections 2024-09-21T00:59:51.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:51.287+0000 7fce7b800640 1 -- 172.21.15.39:0/1280146300 wait complete. 2024-09-21T00:59:51.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:51 smithi150 ceph-mon[34945]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:51.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:51 smithi160 ceph-mon[34696]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:52.087 INFO:tasks.cephadm:Adding mds.a on smithi039 2024-09-21T00:59:52.087 INFO:tasks.cephadm:Adding mds.d on smithi039 2024-09-21T00:59:52.087 INFO:tasks.cephadm:Adding mds.g on smithi039 2024-09-21T00:59:52.087 INFO:tasks.cephadm:Adding mds.j on smithi039 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.b on smithi150 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.e on smithi150 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.h on smithi150 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.k on smithi150 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.c on smithi160 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.f on smithi160 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.i on smithi160 2024-09-21T00:59:52.088 INFO:tasks.cephadm:Adding mds.l on smithi160 2024-09-21T00:59:52.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph orch apply mds all '12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l' 2024-09-21T00:59:52.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:52 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1280146300' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T00:59:52.348 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T00:59:52.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:52 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1280146300' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T00:59:52.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:52 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1280146300' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T00:59:52.849 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 -- 172.21.15.160:0/4254937373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 msgr2=0x7f7fc8103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:52.849 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/4254937373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc8103500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009ca0 tx=0x7f7fb802a8c0 comp rx=0 tx=0).stop 2024-09-21T00:59:52.849 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 -- 172.21.15.160:0/4254937373 shutdown_connections 2024-09-21T00:59:52.849 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/4254937373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105ac0 0x7f7fc810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:52.849 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/4254937373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fc8105170 0x7f7fc8105580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:52.849 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/4254937373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc8103500 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:52.850 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 -- 172.21.15.160:0/4254937373 >> 172.21.15.160:0/4254937373 conn(0x7f7fc8077bc0 msgr2=0x7f7fc8078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:52.850 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 -- 172.21.15.160:0/4254937373 shutdown_connections 2024-09-21T00:59:52.850 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.848+0000 7f7fccc6b640 1 -- 172.21.15.160:0/4254937373 wait complete. 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 Processor -- start 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 -- start start 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105170 0x7f7fc819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fc8105ac0 0x7f7fc819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc8112a40 con 0x7f7fc8105170 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc81128c0 con 0x7f7fc8105ac0 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fccc6b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc8112bc0 con 0x7f7fc81030f0 2024-09-21T00:59:52.852 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fc659c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fc659c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:34076/0 (socket says 172.21.15.160:34076) 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fc659c640 1 -- 172.21.15.160:0/716603988 learned_addr learned my addr 172.21.15.160:0/716603988 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fc6d9d640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fc8105ac0 0x7f7fc819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.849+0000 7f7fc5d9b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105170 0x7f7fc819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc659c640 1 -- 172.21.15.160:0/716603988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fc8105ac0 msgr2=0x7f7fc819bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc659c640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fc8105ac0 0x7f7fc819bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc659c640 1 -- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105170 msgr2=0x7f7fc819b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc659c640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105170 0x7f7fc819b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc659c640 1 -- 172.21.15.160:0/716603988 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc81a65c0 con 0x7f7fc81030f0 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc5d9b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105170 0x7f7fc819b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fc659c640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc819b330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8006010 tx=0x7f7fb80319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:52.853 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8003bf0 con 0x7f7fc81030f0 2024-09-21T00:59:52.854 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb8003d90 con 0x7f7fc81030f0 2024-09-21T00:59:52.854 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc81a6850 con 0x7f7fc81030f0 2024-09-21T00:59:52.854 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb802fc20 con 0x7f7fc81030f0 2024-09-21T00:59:52.854 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.850+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc81a6e10 con 0x7f7fc81030f0 2024-09-21T00:59:52.854 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.851+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc8103500 con 0x7f7fc81030f0 2024-09-21T00:59:52.855 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.853+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f7fb80046a0 con 0x7f7fc81030f0 2024-09-21T00:59:52.855 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.853+0000 7f7fb77fe640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7fb00b6590 0x7f7fb00b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:52.855 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.853+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f7fb81082a0 con 0x7f7fc81030f0 2024-09-21T00:59:52.855 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.854+0000 7f7fc5d9b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7fb00b6590 0x7f7fb00b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:52.855 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.854+0000 7f7fc5d9b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7fb00b6590 0x7f7fb00b8aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0058c0 tx=0x7f7fbc005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:52.858 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:52.858+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7fb80cef00 con 0x7f7fc81030f0 2024-09-21T00:59:53.049 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.048+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l", "target": ["mon-mgr", ""]}) -- 0x7f7fc8063800 con 0x7f7fb00b6590 2024-09-21T00:59:53.062 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mds.all update... 2024-09-21T00:59:53.062 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.061+0000 7f7fb77fe640 1 -- 172.21.15.160:0/716603988 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f7fc8063800 con 0x7f7fb00b6590 2024-09-21T00:59:53.064 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7fb00b6590 msgr2=0x7f7fb00b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7fb00b6590 0x7f7fb00b8aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0058c0 tx=0x7f7fbc005850 comp rx=0 tx=0).stop 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 msgr2=0x7f7fc819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc819b330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8006010 tx=0x7f7fb80319f0 comp rx=0 tx=0).stop 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 shutdown_connections 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7fb00b6590 0x7f7fb00b8aa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fc8105ac0 0x7f7fc819bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fc8105170 0x7f7fc819b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 --2- 172.21.15.160:0/716603988 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fc81030f0 0x7f7fc819b330 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 >> 172.21.15.160:0/716603988 conn(0x7f7fc8077bc0 msgr2=0x7f7fc8106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 shutdown_connections 2024-09-21T00:59:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T00:59:53.064+0000 7f7fccc6b640 1 -- 172.21.15.160:0/716603988 wait complete. 2024-09-21T00:59:53.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:53 smithi160 ceph-mon[34696]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:53 smithi039 ceph-mon[30415]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:53.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:53 smithi150 ceph-mon[34945]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:53.713 DEBUG:teuthology.orchestra.run.smithi039:mds.a> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.a.service 2024-09-21T00:59:53.717 DEBUG:teuthology.orchestra.run.smithi039:mds.d> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.d.service 2024-09-21T00:59:53.761 DEBUG:teuthology.orchestra.run.smithi039:mds.g> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.g.service 2024-09-21T00:59:53.806 DEBUG:teuthology.orchestra.run.smithi039:mds.j> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.j.service 2024-09-21T00:59:53.852 DEBUG:teuthology.orchestra.run.smithi150:mds.b> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.b.service 2024-09-21T00:59:53.855 DEBUG:teuthology.orchestra.run.smithi150:mds.e> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.e.service 2024-09-21T00:59:53.899 DEBUG:teuthology.orchestra.run.smithi150:mds.h> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.h.service 2024-09-21T00:59:53.941 DEBUG:teuthology.orchestra.run.smithi150:mds.k> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.k.service 2024-09-21T00:59:53.984 DEBUG:teuthology.orchestra.run.smithi160:mds.c> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.c.service 2024-09-21T00:59:53.987 DEBUG:teuthology.orchestra.run.smithi160:mds.f> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.f.service 2024-09-21T00:59:54.031 DEBUG:teuthology.orchestra.run.smithi160:mds.i> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.i.service 2024-09-21T00:59:54.074 DEBUG:teuthology.orchestra.run.smithi160:mds.l> sudo journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.l.service 2024-09-21T00:59:54.118 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-21T00:59:54.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T00:59:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='client.24661 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l;count:12 2024-09-21T00:59:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:54.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:54.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:54.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:54.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:54.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:54.319 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='client.24661 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l;count:12 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:54.320 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='client.24661 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l;count:12 2024-09-21T00:59:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T00:59:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T00:59:54.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:54.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:54.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:54.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 -- 172.21.15.39:0/4257731338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a01047d0 msgr2=0x7f42a0104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 --2- 172.21.15.39:0/4257731338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a01047d0 0x7f42a0104be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4290009900 tx=0x7f429002a410 comp rx=0 tx=0).stop 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 -- 172.21.15.39:0/4257731338 shutdown_connections 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 --2- 172.21.15.39:0/4257731338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42a010dc30 0x7f42a0110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 --2- 172.21.15.39:0/4257731338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a0105120 0x7f42a010d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 --2- 172.21.15.39:0/4257731338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a01047d0 0x7f42a0104be0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 -- 172.21.15.39:0/4257731338 >> 172.21.15.39:0/4257731338 conn(0x7f42a00fe2d0 msgr2=0x7f42a0100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 -- 172.21.15.39:0/4257731338 shutdown_connections 2024-09-21T00:59:55.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.145+0000 7f429f200640 1 -- 172.21.15.39:0/4257731338 wait complete. 2024-09-21T00:59:55.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.146+0000 7f429f200640 1 Processor -- start 2024-09-21T00:59:55.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.146+0000 7f429f200640 1 -- start start 2024-09-21T00:59:55.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a01047d0 0x7f42a0102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:55.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a0105120 0x7f42a0102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42a010dc30 0x7f42a0102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f42a0113900 con 0x7f42a01047d0 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f42a0113780 con 0x7f42a010dc30 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f42a0113a80 con 0x7f42a0105120 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a01047d0 0x7f42a0102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429d400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a0105120 0x7f42a0102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a01047d0 0x7f42a0102510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48214/0 (socket says 172.21.15.39:48214) 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429de00640 1 -- 172.21.15.39:0/911687009 learned_addr learned my addr 172.21.15.39:0/911687009 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429d400640 1 -- 172.21.15.39:0/911687009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42a010dc30 msgr2=0x7f42a0102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429d400640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42a010dc30 0x7f42a0102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429d400640 1 -- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a01047d0 msgr2=0x7f42a0102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429d400640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a01047d0 0x7f42a0102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.147+0000 7f429d400640 1 -- 172.21.15.39:0/911687009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a01a85f0 con 0x7f42a0105120 2024-09-21T00:59:55.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.148+0000 7f429d400640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a0105120 0x7f42a0102a50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f42a006b4a0 tx=0x7f428c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:55.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.148+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c00cc40 con 0x7f42a0105120 2024-09-21T00:59:55.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.148+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f428c00cde0 con 0x7f42a0105120 2024-09-21T00:59:55.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.148+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c018780 con 0x7f42a0105120 2024-09-21T00:59:55.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.148+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a01a88e0 con 0x7f42a0105120 2024-09-21T00:59:55.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.148+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a01a8dd0 con 0x7f42a0105120 2024-09-21T00:59:55.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.149+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a0104be0 con 0x7f42a0105120 2024-09-21T00:59:55.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.151+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f428c010a90 con 0x7f42a0105120 2024-09-21T00:59:55.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.153+0000 7f4296a00640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f42840b65e0 0x7f42840b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:55.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.153+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f428c0e77d0 con 0x7f42a0105120 2024-09-21T00:59:55.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.153+0000 7f429de00640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f42840b65e0 0x7f42840b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:55.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.154+0000 7f429de00640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f42840b65e0 0x7f42840b8af0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4290005ec0 tx=0x7f4290005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:55.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.155+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f428c0ae390 con 0x7f42a0105120 2024-09-21T00:59:55.333 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:55 smithi160 ceph-mon[34696]: Deploying daemon mds.l on smithi160 2024-09-21T00:59:55.333 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:55 smithi160 ceph-mon[34696]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:55.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.392+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42a0103810 con 0x7f42a0105120 2024-09-21T00:59:55.393 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:55 smithi039 ceph-mon[30415]: Deploying daemon mds.l on smithi160 2024-09-21T00:59:55.393 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:55 smithi039 ceph-mon[30415]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:55.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.393+0000 7f4296a00640 1 -- 172.21.15.39:0/911687009 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f428c0b2770 con 0x7f42a0105120 2024-09-21T00:59:55.394 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:59:55.394 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":1,"btime":"2024-09-21T00:51:02:310817+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-21T00:59:55.394 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 1 2024-09-21T00:59:55.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f42840b65e0 msgr2=0x7f42840b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f42840b65e0 0x7f42840b8af0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4290005ec0 tx=0x7f4290005a20 comp rx=0 tx=0).stop 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a0105120 msgr2=0x7f42a0102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a0105120 0x7f42a0102a50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f42a006b4a0 tx=0x7f428c00ee50 comp rx=0 tx=0).stop 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 shutdown_connections 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f42840b65e0 0x7f42840b8af0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42a010dc30 0x7f42a0102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42a0105120 0x7f42a0102a50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 --2- 172.21.15.39:0/911687009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42a01047d0 0x7f42a0102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 >> 172.21.15.39:0/911687009 conn(0x7f42a00fe2d0 msgr2=0x7f42a01076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.396+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 shutdown_connections 2024-09-21T00:59:55.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:55.397+0000 7f429f200640 1 -- 172.21.15.39:0/911687009 wait complete. 2024-09-21T00:59:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:55 smithi150 ceph-mon[34945]: Deploying daemon mds.l on smithi160 2024-09-21T00:59:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:55 smithi150 ceph-mon[34945]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:55.607 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:55 smithi160 systemd[1]: Starting Ceph mds.l for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:59:56.022 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:55 smithi160 podman[63683]: 2024-09-21 00:59:55.629761348 +0000 UTC m=+0.016029885 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T00:59:56.022 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:55 smithi160 podman[63683]: 2024-09-21 00:59:55.75204114 +0000 UTC m=+0.138309716 container create c202ad70b51127b9ee720c7ed9ffdf1522b437ebcfb8fc3495a214f8d71c9d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T00:59:56.022 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:56 smithi160 podman[63683]: 2024-09-21 00:59:56.021862699 +0000 UTC m=+0.408131255 container init c202ad70b51127b9ee720c7ed9ffdf1522b437ebcfb8fc3495a214f8d71c9d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T00:59:56.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/911687009' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T00:59:56.357 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:56 smithi160 podman[63683]: 2024-09-21 00:59:56.025314698 +0000 UTC m=+0.411583241 container start c202ad70b51127b9ee720c7ed9ffdf1522b437ebcfb8fc3495a214f8d71c9d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T00:59:56.357 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:56 smithi160 ceph-mds[63702]: -- 172.21.15.160:0/770984996 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f728577860 con 0x55f72c71f600 2024-09-21T00:59:56.357 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l[63698]: starting mds.l at 2024-09-21T00:59:56.357 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:56 smithi160 bash[63683]: c202ad70b51127b9ee720c7ed9ffdf1522b437ebcfb8fc3495a214f8d71c9d1b 2024-09-21T00:59:56.357 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 00:59:56 smithi160 systemd[1]: Started Ceph mds.l for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:59:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/911687009' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T00:59:56.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/911687009' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6832/166386963,v1:172.21.15.160:6833/166386963] up:boot 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: fsmap 1 up:standby 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:57.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:57 smithi150 ceph-mon[34945]: Deploying daemon mds.k on smithi150 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6832/166386963,v1:172.21.15.160:6833/166386963] up:boot 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: fsmap 1 up:standby 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:57 smithi160 ceph-mon[34696]: Deploying daemon mds.k on smithi150 2024-09-21T00:59:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6832/166386963,v1:172.21.15.160:6833/166386963] up:boot 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: fsmap 1 up:standby 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:57.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:57 smithi039 ceph-mon[30415]: Deploying daemon mds.k on smithi150 2024-09-21T00:59:58.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T00:59:58.314 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 systemd[1]: Starting Ceph mds.k for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T00:59:58.589 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T00:59:58.648 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 podman[63596]: 2024-09-21 00:59:58.292053233 +0000 UTC m=+0.015680478 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T00:59:58.648 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 podman[63596]: 2024-09-21 00:59:58.494691246 +0000 UTC m=+0.218318515 container create b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T00:59:59.064 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 podman[63596]: 2024-09-21 00:59:58.755554229 +0000 UTC m=+0.479181455 container init b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:59:59.064 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 podman[63596]: 2024-09-21 00:59:58.758377481 +0000 UTC m=+0.482004708 container start b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:59:59.064 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 ceph-mds[63615]: -- 172.21.15.150:0/262212575 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56537385d860 con 0x565377a05600 2024-09-21T00:59:59.064 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k[63611]: starting mds.k at 2024-09-21T00:59:59.064 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 bash[63596]: b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc 2024-09-21T00:59:59.064 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 00:59:58 smithi150 systemd[1]: Started Ceph mds.k for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T00:59:59.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 -- 172.21.15.39:0/2508662821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f2810c010 msgr2=0x7f7f28112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:59.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 --2- 172.21.15.39:0/2508662821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f2810c010 0x7f7f28112840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009ca0 tx=0x7f7f2002a8c0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 -- 172.21.15.39:0/2508662821 shutdown_connections 2024-09-21T00:59:59.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 --2- 172.21.15.39:0/2508662821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f2810c010 0x7f7f28112840 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 --2- 172.21.15.39:0/2508662821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f2810b6c0 0x7f7f2810bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 --2- 172.21.15.39:0/2508662821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f28105d80 0x7f7f28106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.262+0000 7f7f2e200640 1 -- 172.21.15.39:0/2508662821 >> 172.21.15.39:0/2508662821 conn(0x7f7f281017d0 msgr2=0x7f7f28103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:59.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.263+0000 7f7f2e200640 1 -- 172.21.15.39:0/2508662821 shutdown_connections 2024-09-21T00:59:59.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.263+0000 7f7f2e200640 1 -- 172.21.15.39:0/2508662821 wait complete. 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 Processor -- start 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 -- start start 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f28105d80 0x7f7f2819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 0x7f7f2819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f2810c010 0x7f7f281a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28115d20 con 0x7f7f28105d80 2024-09-21T00:59:59.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28115ba0 con 0x7f7f2810c010 2024-09-21T00:59:59.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.264+0000 7f7f2e200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28115ea0 con 0x7f7f2810b6c0 2024-09-21T00:59:59.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f1fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 0x7f7f2819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:59.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f28105d80 0x7f7f2819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:59.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f1fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 0x7f7f2819cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53424/0 (socket says 172.21.15.39:53424) 2024-09-21T00:59:59.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f1fe00640 1 -- 172.21.15.39:0/3402371624 learned_addr learned my addr 172.21.15.39:0/3402371624 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T00:59:59.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27e00640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f2810c010 0x7f7f281a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:59.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27400640 1 -- 172.21.15.39:0/3402371624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 msgr2=0x7f7f2819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:59.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27400640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 0x7f7f2819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27400640 1 -- 172.21.15.39:0/3402371624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f2810c010 msgr2=0x7f7f281a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:59.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27400640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f2810c010 0x7f7f281a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f27400640 1 -- 172.21.15.39:0/3402371624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f281a25a0 con 0x7f7f28105d80 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.265+0000 7f7f1fe00640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 0x7f7f2819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.266+0000 7f7f27400640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f28105d80 0x7f7f2819c630 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7f18005fd0 tx=0x7f7f1800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.266+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18009c30 con 0x7f7f28105d80 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.266+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f281a2890 con 0x7f7f28105d80 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.266+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f18009dd0 con 0x7f7f28105d80 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.266+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18010460 con 0x7f7f28105d80 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.266+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f281a8ff0 con 0x7f7f28105d80 2024-09-21T00:59:59.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.267+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f28106190 con 0x7f7f28105d80 2024-09-21T00:59:59.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.270+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f7f18003e80 con 0x7f7f28105d80 2024-09-21T00:59:59.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.271+0000 7f7f24c00640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7f0c0b65e0 0x7f7f0c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T00:59:59.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.272+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f7f180e6b60 con 0x7f7f28105d80 2024-09-21T00:59:59.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.272+0000 7f7f1fe00640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7f0c0b65e0 0x7f7f0c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T00:59:59.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.273+0000 7f7f1fe00640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7f0c0b65e0 0x7f7f0c0b8af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f100095f0 tx=0x7f7f10005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T00:59:59.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.275+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7f18017020 con 0x7f7f28105d80 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 00:59:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:59.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 00:59:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T00:59:59.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.532+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f2819d760 con 0x7f7f28105d80 2024-09-21T00:59:59.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.533+0000 7f7f24c00640 1 -- 172.21.15.39:0/3402371624 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2260 (secure 0 0 0) 0x7f7f180b1b80 con 0x7f7f28105d80 2024-09-21T00:59:59.533 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T00:59:59.533 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":3,"btime":"2024-09-21T00:59:59:080910+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T00:59:59.533 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 3 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7f0c0b65e0 msgr2=0x7f7f0c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7f0c0b65e0 0x7f7f0c0b8af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f100095f0 tx=0x7f7f10005f90 comp rx=0 tx=0).stop 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f28105d80 msgr2=0x7f7f2819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f28105d80 0x7f7f2819c630 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7f18005fd0 tx=0x7f7f1800dde0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 shutdown_connections 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7f0c0b65e0 0x7f7f0c0b8af0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f2810c010 0x7f7f281a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7f2810b6c0 0x7f7f2819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 --2- 172.21.15.39:0/3402371624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f28105d80 0x7f7f2819c630 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T00:59:59.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 >> 172.21.15.39:0/3402371624 conn(0x7f7f281017d0 msgr2=0x7f7f28108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T00:59:59.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 shutdown_connections 2024-09-21T00:59:59.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T00:59:59.535+0000 7f7f2e200640 1 -- 172.21.15.39:0/3402371624 wait complete. 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T00:59:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 00:59:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:00.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:00 smithi160 ceph-mon[34696]: Deploying daemon mds.j on smithi039 2024-09-21T01:00:00.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:00 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6832/845239112,v1:172.21.15.150:6833/845239112] up:boot 2024-09-21T01:00:00.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:00 smithi160 ceph-mon[34696]: fsmap 2 up:standby 2024-09-21T01:00:00.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T01:00:00.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:00 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3402371624' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:00.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:00 smithi160 ceph-mon[34696]: overall HEALTH_OK 2024-09-21T01:00:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:00 smithi039 ceph-mon[30415]: Deploying daemon mds.j on smithi039 2024-09-21T01:00:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:00 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6832/845239112,v1:172.21.15.150:6833/845239112] up:boot 2024-09-21T01:00:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:00 smithi039 ceph-mon[30415]: fsmap 2 up:standby 2024-09-21T01:00:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T01:00:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:00 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3402371624' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:00 smithi039 ceph-mon[30415]: overall HEALTH_OK 2024-09-21T01:00:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:00 smithi150 ceph-mon[34945]: Deploying daemon mds.j on smithi039 2024-09-21T01:00:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:00 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6832/845239112,v1:172.21.15.150:6833/845239112] up:boot 2024-09-21T01:00:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:00 smithi150 ceph-mon[34945]: fsmap 2 up:standby 2024-09-21T01:00:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T01:00:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:00 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3402371624' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:00 smithi150 ceph-mon[34945]: overall HEALTH_OK 2024-09-21T01:00:01.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:01 smithi039 ceph-mon[30415]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:01.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:01 smithi150 ceph-mon[34945]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:01.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:01 smithi160 ceph-mon[34696]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:02.075 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:01 smithi039 systemd[1]: Starting Ceph mds.j for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:00:02.365 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 podman[70900]: 2024-09-21 01:00:01.978508959 +0000 UTC m=+0.028205922 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:00:02.365 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 podman[70900]: 2024-09-21 01:00:02.119407691 +0000 UTC m=+0.169104642 container create 5c49624a179c29775823354c27798b6b61afdc2e0054d1c54740e1851345e5f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T01:00:02.365 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 podman[70900]: 2024-09-21 01:00:02.364817768 +0000 UTC m=+0.414514719 container init 5c49624a179c29775823354c27798b6b61afdc2e0054d1c54740e1851345e5f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T01:00:02.653 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 podman[70900]: 2024-09-21 01:00:02.371184143 +0000 UTC m=+0.420881095 container start 5c49624a179c29775823354c27798b6b61afdc2e0054d1c54740e1851345e5f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-21T01:00:02.653 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 ceph-mds[70918]: -- 172.21.15.39:0/3749731487 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557945609860 con 0x5579497b1a80 2024-09-21T01:00:02.653 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j[70914]: starting mds.j at 2024-09-21T01:00:02.653 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 bash[70900]: 5c49624a179c29775823354c27798b6b61afdc2e0054d1c54740e1851345e5f9 2024-09-21T01:00:02.654 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:00:02 smithi039 systemd[1]: Started Ceph mds.j for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:03.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: Deploying daemon mds.d on smithi039 2024-09-21T01:00:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.39:6834/319446739,v1:172.21.15.39:6835/319446739] up:boot 2024-09-21T01:00:03.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: fsmap 3 up:standby 2024-09-21T01:00:03.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:03 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T01:00:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: Deploying daemon mds.d on smithi039 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.39:6834/319446739,v1:172.21.15.39:6835/319446739] up:boot 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: fsmap 3 up:standby 2024-09-21T01:00:03.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:03 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: Deploying daemon mds.d on smithi039 2024-09-21T01:00:04.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.39:6834/319446739,v1:172.21.15.39:6835/319446739] up:boot 2024-09-21T01:00:04.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: fsmap 3 up:standby 2024-09-21T01:00:04.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:03 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T01:00:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:05.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 -- 172.21.15.39:0/908510759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25541094a0 msgr2=0x7f25541098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:05.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 --2- 172.21.15.39:0/908510759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25541094a0 0x7f25541098b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2540009900 tx=0x7f254002a410 comp rx=0 tx=0).stop 2024-09-21T01:00:05.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 -- 172.21.15.39:0/908510759 shutdown_connections 2024-09-21T01:00:05.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 --2- 172.21.15.39:0/908510759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2554109df0 0x7f2554110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 --2- 172.21.15.39:0/908510759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25541094a0 0x7f25541098b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 --2- 172.21.15.39:0/908510759 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2554103b60 0x7f2554103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 -- 172.21.15.39:0/908510759 >> 172.21.15.39:0/908510759 conn(0x7f255407b850 msgr2=0x7f255407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:05.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 -- 172.21.15.39:0/908510759 shutdown_connections 2024-09-21T01:00:05.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.159+0000 7f255b800640 1 -- 172.21.15.39:0/908510759 wait complete. 2024-09-21T01:00:05.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.160+0000 7f255b800640 1 Processor -- start 2024-09-21T01:00:05.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.160+0000 7f255b800640 1 -- start start 2024-09-21T01:00:05.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f255b800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2554103b60 0x7f255419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:05.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f255b800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 0x7f255419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:05.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f255b800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2554109df0 0x7f255419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:05.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f255b800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2554113b00 con 0x7f2554103b60 2024-09-21T01:00:05.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f255b800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2554113980 con 0x7f25541094a0 2024-09-21T01:00:05.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f255b800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2554113c80 con 0x7f2554109df0 2024-09-21T01:00:05.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f2553e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 0x7f255419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:05.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f2553e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 0x7f255419aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35830/0 (socket says 172.21.15.39:35830) 2024-09-21T01:00:05.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f2553e00640 1 -- 172.21.15.39:0/415268948 learned_addr learned my addr 172.21.15.39:0/415268948 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:05.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f2558e00640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2554103b60 0x7f255419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:05.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.161+0000 7f2559800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2554109df0 0x7f255419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:05.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.162+0000 7f2553e00640 1 -- 172.21.15.39:0/415268948 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2554109df0 msgr2=0x7f255419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.162+0000 7f2553e00640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2554109df0 0x7f255419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.162+0000 7f2553e00640 1 -- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2554103b60 msgr2=0x7f255419a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.162+0000 7f2553e00640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2554103b60 0x7f255419a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.162+0000 7f2553e00640 1 -- 172.21.15.39:0/415268948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25541a03e0 con 0x7f25541094a0 2024-09-21T01:00:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.162+0000 7f2553e00640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 0x7f255419aa10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2540005ab0 tx=0x7f2540003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.163+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254003f070 con 0x7f25541094a0 2024-09-21T01:00:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.163+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2540031b50 con 0x7f25541094a0 2024-09-21T01:00:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.163+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25541a0670 con 0x7f25541094a0 2024-09-21T01:00:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.163+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540035030 con 0x7f25541094a0 2024-09-21T01:00:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.163+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25541a6e30 con 0x7f25541094a0 2024-09-21T01:00:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.164+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2554104de0 con 0x7f25541094a0 2024-09-21T01:00:05.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.167+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2540042070 con 0x7f25541094a0 2024-09-21T01:00:05.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.168+0000 7f2551600640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f253c0b6560 0x7f253c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:05.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.168+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f25401079d0 con 0x7f25541094a0 2024-09-21T01:00:05.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.169+0000 7f2558e00640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f253c0b6560 0x7f253c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:05.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.169+0000 7f2558e00640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f253c0b6560 0x7f253c0b8a70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2548009bd0 tx=0x7f2548005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:05.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.170+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f25400d2970 con 0x7f25541094a0 2024-09-21T01:00:05.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.417+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2554103f70 con 0x7f25541094a0 2024-09-21T01:00:05.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.418+0000 7f2551600640 1 -- 172.21.15.39:0/415268948 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3082 (secure 0 0 0) 0x7f2554103f70 con 0x7f25541094a0 2024-09-21T01:00:05.419 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:05.419 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":4,"btime":"2024-09-21T01:00:03:198236+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T01:00:05.419 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 4 2024-09-21T01:00:05.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f253c0b6560 msgr2=0x7f253c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:05.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f253c0b6560 0x7f253c0b8a70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2548009bd0 tx=0x7f2548005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:05.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 msgr2=0x7f255419aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:05.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 0x7f255419aa10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2540005ab0 tx=0x7f2540003b90 comp rx=0 tx=0).stop 2024-09-21T01:00:05.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 shutdown_connections 2024-09-21T01:00:05.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2554109df0 0x7f255419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f253c0b6560 0x7f253c0b8a70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25541094a0 0x7f255419aa10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 --2- 172.21.15.39:0/415268948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2554103b60 0x7f255419a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:05.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 >> 172.21.15.39:0/415268948 conn(0x7f255407b850 msgr2=0x7f25541054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:05.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 shutdown_connections 2024-09-21T01:00:05.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:05.421+0000 7f255b800640 1 -- 172.21.15.39:0/415268948 wait complete. 2024-09-21T01:00:05.670 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:05 smithi039 ceph-mon[30415]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:05.670 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:05 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/415268948' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:05 smithi160 ceph-mon[34696]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:05 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/415268948' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:05 smithi150 ceph-mon[34945]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:05 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/415268948' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:06.270 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:00:06 smithi039 bash[71269]: 7e6a80933bc923b14e6f7b437f892df2b419086981d32f4ed908914e44ba9231 2024-09-21T01:00:06.544 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:00:06 smithi039 systemd[1]: Started Ceph mds.d for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: Deploying daemon mds.e on smithi150 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.39:6836/1544048762,v1:172.21.15.39:6837/1544048762] up:boot 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: fsmap 4 up:standby 2024-09-21T01:00:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T01:00:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: Deploying daemon mds.e on smithi150 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.39:6836/1544048762,v1:172.21.15.39:6837/1544048762] up:boot 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: fsmap 4 up:standby 2024-09-21T01:00:07.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: Deploying daemon mds.e on smithi150 2024-09-21T01:00:07.776 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.39:6836/1544048762,v1:172.21.15.39:6837/1544048762] up:boot 2024-09-21T01:00:07.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: fsmap 4 up:standby 2024-09-21T01:00:07.777 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T01:00:08.498 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 systemd[1]: Starting Ceph mds.e for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:00:08.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:08.810 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 podman[63880]: 2024-09-21 01:00:08.400113797 +0000 UTC m=+0.016163357 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:00:08.810 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 podman[63880]: 2024-09-21 01:00:08.515733081 +0000 UTC m=+0.131782689 container create 11c0020994b70a053eb21bb8d698b46b6bf8be9adf423d3629d5948aaf3c0dcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T01:00:08.810 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 podman[63880]: 2024-09-21 01:00:08.810124205 +0000 UTC m=+0.426173764 container init 11c0020994b70a053eb21bb8d698b46b6bf8be9adf423d3629d5948aaf3c0dcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:00:09.044 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:09.063 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 podman[63880]: 2024-09-21 01:00:08.813008354 +0000 UTC m=+0.429057916 container start 11c0020994b70a053eb21bb8d698b46b6bf8be9adf423d3629d5948aaf3c0dcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:00:09.064 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 ceph-mds[63899]: -- 172.21.15.150:0/979248802 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616bf339860 con 0x5616c34e1180 2024-09-21T01:00:09.064 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e[63895]: starting mds.e at 2024-09-21T01:00:09.064 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 bash[63880]: 11c0020994b70a053eb21bb8d698b46b6bf8be9adf423d3629d5948aaf3c0dcb 2024-09-21T01:00:09.064 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:00:08 smithi150 systemd[1]: Started Ceph mds.e for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:09.520 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:09.598 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:09.598 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.599 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.599 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.599 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:09.599 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:09.599 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:09.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 -- 172.21.15.39:0/3484605436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 msgr2=0x7f6c10103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:09.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 --2- 172.21.15.39:0/3484605436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c10103f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009900 tx=0x7f6c0402a410 comp rx=0 tx=0).stop 2024-09-21T01:00:09.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 -- 172.21.15.39:0/3484605436 shutdown_connections 2024-09-21T01:00:09.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 --2- 172.21.15.39:0/3484605436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c10109df0 0x7f6c10110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 --2- 172.21.15.39:0/3484605436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 0x7f6c101098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 --2- 172.21.15.39:0/3484605436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c10103f70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 -- 172.21.15.39:0/3484605436 >> 172.21.15.39:0/3484605436 conn(0x7f6c1007b850 msgr2=0x7f6c1007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:09.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 -- 172.21.15.39:0/3484605436 shutdown_connections 2024-09-21T01:00:09.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.698+0000 7f6c17800640 1 -- 172.21.15.39:0/3484605436 wait complete. 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.699+0000 7f6c17800640 1 Processor -- start 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.699+0000 7f6c17800640 1 -- start start 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c17800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c1019a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c17800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 0x7f6c1019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c17800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c10109df0 0x7f6c1019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c17800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10113b00 con 0x7f6c10103b60 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c17800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10113980 con 0x7f6c10109df0 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c17800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10113c80 con 0x7f6c101094a0 2024-09-21T01:00:09.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c1019a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:09.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c0fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 0x7f6c1019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:09.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c1019a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57334/0 (socket says 172.21.15.39:57334) 2024-09-21T01:00:09.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 -- 172.21.15.39:0/3511760718 learned_addr learned my addr 172.21.15.39:0/3511760718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:09.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c15800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c10109df0 0x7f6c1019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 -- 172.21.15.39:0/3511760718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 msgr2=0x7f6c1019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 0x7f6c1019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 -- 172.21.15.39:0/3511760718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c10109df0 msgr2=0x7f6c1019fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c10109df0 0x7f6c1019fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c14e00640 1 -- 172.21.15.39:0/3511760718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c101a04e0 con 0x7f6c10103b60 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.700+0000 7f6c0fe00640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 0x7f6c1019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.701+0000 7f6c14e00640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c1019a570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c04005ab0 tx=0x7f6c04003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.701+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0403f070 con 0x7f6c10103b60 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.701+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c04031d60 con 0x7f6c10103b60 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.701+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c101a0770 con 0x7f6c10103b60 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.701+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04035030 con 0x7f6c10103b60 2024-09-21T01:00:09.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.701+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c101a6f30 con 0x7f6c10103b60 2024-09-21T01:00:09.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.702+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c10103f70 con 0x7f6c10103b60 2024-09-21T01:00:09.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.705+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f6c04042070 con 0x7f6c10103b60 2024-09-21T01:00:09.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.707+0000 7f6c0d600640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6bf80b6560 0x7f6bf80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:09.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.707+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f6c04107e60 con 0x7f6c10103b60 2024-09-21T01:00:09.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.708+0000 7f6c0fe00640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6bf80b6560 0x7f6bf80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:09.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.708+0000 7f6c0fe00640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6bf80b6560 0x7f6bf80b8a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6c10109200 tx=0x7f6bfc00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:09.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.710+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6c040d2e00 con 0x7f6c10103b60 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:09.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.948+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c1010bbf0 con 0x7f6c10103b60 2024-09-21T01:00:09.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.948+0000 7f6c0d600640 1 -- 172.21.15.39:0/3511760718 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4735 (secure 0 0 0) 0x7f6c040d7d40 con 0x7f6c10103b60 2024-09-21T01:00:09.950 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:09.950 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":6,"btime":"2024-09-21T01:00:09:343913+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-09-21T01:00:09.950 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 6 2024-09-21T01:00:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.950+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6bf80b6560 msgr2=0x7f6bf80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.950+0000 7f6c17800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6bf80b6560 0x7f6bf80b8a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6c10109200 tx=0x7f6bfc00b390 comp rx=0 tx=0).stop 2024-09-21T01:00:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 msgr2=0x7f6c1019a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c1019a570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c04005ab0 tx=0x7f6c04003cd0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 shutdown_connections 2024-09-21T01:00:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6bf80b6560 0x7f6bf80b8a70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c10109df0 0x7f6c1019fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6c101094a0 0x7f6c1019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 --2- 172.21.15.39:0/3511760718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c10103b60 0x7f6c1019a570 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 >> 172.21.15.39:0/3511760718 conn(0x7f6c1007b850 msgr2=0x7f6c10107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 shutdown_connections 2024-09-21T01:00:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:09.951+0000 7f6c17800640 1 -- 172.21.15.39:0/3511760718 wait complete. 2024-09-21T01:00:10.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:10 smithi039 ceph-mon[30415]: Deploying daemon mds.f on smithi160 2024-09-21T01:00:10.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:10 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6834/2170415276,v1:172.21.15.150:6835/2170415276] up:boot 2024-09-21T01:00:10.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:10 smithi039 ceph-mon[30415]: fsmap 5 up:standby 2024-09-21T01:00:10.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T01:00:10.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:10 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3511760718' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:10.656 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:10 smithi160 ceph-mon[34696]: Deploying daemon mds.f on smithi160 2024-09-21T01:00:10.656 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:10 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6834/2170415276,v1:172.21.15.150:6835/2170415276] up:boot 2024-09-21T01:00:10.656 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:10 smithi160 ceph-mon[34696]: fsmap 5 up:standby 2024-09-21T01:00:10.656 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T01:00:10.656 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:10 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3511760718' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:10 smithi150 ceph-mon[34945]: Deploying daemon mds.f on smithi160 2024-09-21T01:00:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:10 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6834/2170415276,v1:172.21.15.150:6835/2170415276] up:boot 2024-09-21T01:00:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:10 smithi150 ceph-mon[34945]: fsmap 5 up:standby 2024-09-21T01:00:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T01:00:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:10 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3511760718' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:11.357 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 podman[63952]: 2024-09-21 01:00:11.24883386 +0000 UTC m=+0.122822278 container create a0e492f5e6194f7c5b7c8c8771ef501cb523742d2ba6f3bc9bc829e99ebac444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T01:00:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:11 smithi039 ceph-mon[30415]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:11.669 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:11 smithi150 ceph-mon[34945]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:11 smithi160 ceph-mon[34696]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 podman[63952]: 2024-09-21 01:00:11.49306848 +0000 UTC m=+0.367056856 container init a0e492f5e6194f7c5b7c8c8771ef501cb523742d2ba6f3bc9bc829e99ebac444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 podman[63952]: 2024-09-21 01:00:11.49585781 +0000 UTC m=+0.369846186 container start a0e492f5e6194f7c5b7c8c8771ef501cb523742d2ba6f3bc9bc829e99ebac444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 ceph-mds[63971]: -- 172.21.15.160:0/3236119298 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d531d7860 con 0x561d5737fa80 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f[63967]: starting mds.f at 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 bash[63952]: a0e492f5e6194f7c5b7c8c8771ef501cb523742d2ba6f3bc9bc829e99ebac444 2024-09-21T01:00:11.857 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:00:11 smithi160 systemd[1]: Started Ceph mds.f for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:12.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6834/2680876155,v1:172.21.15.160:6835/2680876155] up:boot 2024-09-21T01:00:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: fsmap 6 up:standby 2024-09-21T01:00:12.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6834/2680876155,v1:172.21.15.160:6835/2680876155] up:boot 2024-09-21T01:00:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: fsmap 6 up:standby 2024-09-21T01:00:13.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6834/2680876155,v1:172.21.15.160:6835/2680876155] up:boot 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: fsmap 6 up:standby 2024-09-21T01:00:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T01:00:13.255 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:13 smithi160 ceph-mon[34696]: Deploying daemon mds.c on smithi160 2024-09-21T01:00:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:13 smithi160 ceph-mon[34696]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:13 smithi039 ceph-mon[30415]: Deploying daemon mds.c on smithi160 2024-09-21T01:00:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:13 smithi039 ceph-mon[30415]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:13.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 -- 172.21.15.39:0/2169876963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341092f0 msgr2=0x7f2c34109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2169876963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341092f0 0x7f2c34109700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009900 tx=0x7f2c2002a410 comp rx=0 tx=0).stop 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 -- 172.21.15.39:0/2169876963 shutdown_connections 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2169876963 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c34109c40 0x7f2c34110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2169876963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341092f0 0x7f2c34109700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2169876963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c341039b0 0x7f2c34103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 -- 172.21.15.39:0/2169876963 >> 172.21.15.39:0/2169876963 conn(0x7f2c340ff440 msgr2=0x7f2c34101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 -- 172.21.15.39:0/2169876963 shutdown_connections 2024-09-21T01:00:13.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.949+0000 7f2c3a400640 1 -- 172.21.15.39:0/2169876963 wait complete. 2024-09-21T01:00:13.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.950+0000 7f2c3a400640 1 Processor -- start 2024-09-21T01:00:13.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.950+0000 7f2c3a400640 1 -- start start 2024-09-21T01:00:13.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c3a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 0x7f2c3419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:13.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c3a400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c341092f0 0x7f2c3419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:13.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c3a400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c34109c40 0x7f2c341a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c3a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34113950 con 0x7f2c341039b0 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c3a400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341137d0 con 0x7f2c34109c40 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c3a400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34113ad0 con 0x7f2c341092f0 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c39000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 0x7f2c3419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c39000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 0x7f2c3419c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39834/0 (socket says 172.21.15.39:39834) 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c33e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c341092f0 0x7f2c3419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c39000640 1 -- 172.21.15.39:0/2518398494 learned_addr learned my addr 172.21.15.39:0/2518398494 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:13.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.951+0000 7f2c39a00640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c34109c40 0x7f2c341a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:13.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39000640 1 -- 172.21.15.39:0/2518398494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c341092f0 msgr2=0x7f2c3419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:13.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39000640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c341092f0 0x7f2c3419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:13.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39000640 1 -- 172.21.15.39:0/2518398494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c34109c40 msgr2=0x7f2c341a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:13.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39000640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c34109c40 0x7f2c341a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:13.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39000640 1 -- 172.21.15.39:0/2518398494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c341a2490 con 0x7f2c341039b0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39a00640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c34109c40 0x7f2c341a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c33e00640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c341092f0 0x7f2c3419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c39000640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 0x7f2c3419c520 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800e9f0 tx=0x7f2c2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2800cd80 con 0x7f2c341039b0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c280108e0 con 0x7f2c341039b0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28018980 con 0x7f2c341039b0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.952+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c341a2780 con 0x7f2c341039b0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.953+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c341a8ee0 con 0x7f2c341039b0 2024-09-21T01:00:13.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.954+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c34103dc0 con 0x7f2c341039b0 2024-09-21T01:00:13.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.956+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2c28010430 con 0x7f2c341039b0 2024-09-21T01:00:13.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.957+0000 7f2c31600640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2c1c0b65e0 0x7f2c1c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:13.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.958+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f2c280e7820 con 0x7f2c341039b0 2024-09-21T01:00:13.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.958+0000 7f2c33e00640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2c1c0b65e0 0x7f2c1c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:13.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.958+0000 7f2c33e00640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2c1c0b65e0 0x7f2c1c0b8af0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2c2002aa00 tx=0x7f2c20005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:13.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:13.960+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2c280ae3e0 con 0x7f2c341039b0 2024-09-21T01:00:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:13 smithi150 ceph-mon[34945]: Deploying daemon mds.c on smithi160 2024-09-21T01:00:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:13 smithi150 ceph-mon[34945]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.196+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c341a9180 con 0x7f2c341039b0 2024-09-21T01:00:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.197+0000 7f2c31600640 1 -- 172.21.15.39:0/2518398494 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5563 (secure 0 0 0) 0x7f2c280b27c0 con 0x7f2c341039b0 2024-09-21T01:00:14.198 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:14.199 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"btime":"2024-09-21T01:00:12:368938+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-09-21T01:00:14.199 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-09-21T01:00:14.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.199+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2c1c0b65e0 msgr2=0x7f2c1c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:14.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.199+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2c1c0b65e0 0x7f2c1c0b8af0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2c2002aa00 tx=0x7f2c20005a20 comp rx=0 tx=0).stop 2024-09-21T01:00:14.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.199+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 msgr2=0x7f2c3419c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.199+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 0x7f2c3419c520 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800e9f0 tx=0x7f2c2800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.199+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 shutdown_connections 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.199+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2c1c0b65e0 0x7f2c1c0b8af0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.200+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c34109c40 0x7f2c341a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.200+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2c341092f0 0x7f2c3419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.200+0000 7f2c3a400640 1 --2- 172.21.15.39:0/2518398494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c341039b0 0x7f2c3419c520 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:14.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.200+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 >> 172.21.15.39:0/2518398494 conn(0x7f2c340ff440 msgr2=0x7f2c341068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:14.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.200+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 shutdown_connections 2024-09-21T01:00:14.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:14.200+0000 7f2c3a400640 1 -- 172.21.15.39:0/2518398494 wait complete. 2024-09-21T01:00:14.607 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:00:14 smithi160 systemd[1]: Started Ceph mds.c for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2518398494' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:14.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:14 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2518398494' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:14.931 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:14 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:15.112 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2518398494' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:15.112 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:15.112 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:15.112 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:15.112 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:15.113 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:15.113 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:14 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:15.981 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:15 smithi150 ceph-mon[34945]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:15.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:15 smithi150 ceph-mon[34945]: Deploying daemon mds.h on smithi150 2024-09-21T01:00:15.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:15 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6836/1965195431,v1:172.21.15.160:6837/1965195431] up:boot 2024-09-21T01:00:15.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:15 smithi150 ceph-mon[34945]: fsmap 7 up:standby 2024-09-21T01:00:15.982 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:15 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T01:00:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:15 smithi160 ceph-mon[34696]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:15 smithi160 ceph-mon[34696]: Deploying daemon mds.h on smithi150 2024-09-21T01:00:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:15 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6836/1965195431,v1:172.21.15.160:6837/1965195431] up:boot 2024-09-21T01:00:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:15 smithi160 ceph-mon[34696]: fsmap 7 up:standby 2024-09-21T01:00:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:15 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T01:00:16.117 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:15 smithi039 ceph-mon[30415]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:16.117 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:15 smithi039 ceph-mon[30415]: Deploying daemon mds.h on smithi150 2024-09-21T01:00:16.117 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:15 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6836/1965195431,v1:172.21.15.160:6837/1965195431] up:boot 2024-09-21T01:00:16.117 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:15 smithi039 ceph-mon[30415]: fsmap 7 up:standby 2024-09-21T01:00:16.118 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:15 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T01:00:16.314 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 systemd[1]: Starting Ceph mds.h for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:00:16.665 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 podman[64149]: 2024-09-21 01:00:16.272729213 +0000 UTC m=+0.015844701 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:00:16.666 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 podman[64149]: 2024-09-21 01:00:16.396357291 +0000 UTC m=+0.139472807 container create 2e6bdd1c228035ba75de9dad8d2bf787373de26abb5fa5b8c8c8781cec1268e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:00:16.666 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 podman[64149]: 2024-09-21 01:00:16.665552409 +0000 UTC m=+0.408667896 container init 2e6bdd1c228035ba75de9dad8d2bf787373de26abb5fa5b8c8c8781cec1268e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T01:00:16.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:16 smithi039 ceph-mon[30415]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:16.923 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:17.008 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:16 smithi160 ceph-mon[34696]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:17.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:16 smithi150 ceph-mon[34945]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:17.064 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 podman[64149]: 2024-09-21 01:00:16.669041563 +0000 UTC m=+0.412157041 container start 2e6bdd1c228035ba75de9dad8d2bf787373de26abb5fa5b8c8c8781cec1268e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T01:00:17.064 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 ceph-mds[64169]: -- 172.21.15.150:0/2473731495 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e2ac7b3860 con 0x55e2b095ba80 2024-09-21T01:00:17.064 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h[64165]: starting mds.h at 2024-09-21T01:00:17.065 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 bash[64149]: 2e6bdd1c228035ba75de9dad8d2bf787373de26abb5fa5b8c8c8781cec1268e3 2024-09-21T01:00:17.065 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:00:16 smithi150 systemd[1]: Started Ceph mds.h for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:17.315 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: Deploying daemon mds.i on smithi160 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6836/555411539,v1:172.21.15.150:6837/555411539] up:boot 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: fsmap 8 up:standby 2024-09-21T01:00:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:17 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 -- 172.21.15.39:0/2679953552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d0102ff0 msgr2=0x7f31d010fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 --2- 172.21.15.39:0/2679953552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d0102ff0 0x7f31d010fd10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009900 tx=0x7f31b802a480 comp rx=0 tx=0).stop 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 -- 172.21.15.39:0/2679953552 shutdown_connections 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 --2- 172.21.15.39:0/2679953552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d0110250 0x7f31d0112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 --2- 172.21.15.39:0/2679953552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d0102ff0 0x7f31d010fd10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 --2- 172.21.15.39:0/2679953552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d01026a0 0x7f31d0102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 -- 172.21.15.39:0/2679953552 >> 172.21.15.39:0/2679953552 conn(0x7f31d0079650 msgr2=0x7f31d0079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:18.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 -- 172.21.15.39:0/2679953552 shutdown_connections 2024-09-21T01:00:18.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.007+0000 7f31d5a00640 1 -- 172.21.15.39:0/2679953552 wait complete. 2024-09-21T01:00:18.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.008+0000 7f31d5a00640 1 Processor -- start 2024-09-21T01:00:18.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.008+0000 7f31d5a00640 1 -- start start 2024-09-21T01:00:18.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31d5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d01026a0 0x7f31d019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31d5a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 0x7f31d019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31d5a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d0110250 0x7f31d01a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31d5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0115de0 con 0x7f31d01026a0 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31d5a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0115c60 con 0x7f31d0110250 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31d5a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0115f60 con 0x7f31d0102ff0 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 0x7f31d019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31c7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 0x7f31d019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37598/0 (socket says 172.21.15.39:37598) 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31cea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d01026a0 0x7f31d019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31c7e00640 1 -- 172.21.15.39:0/1846551983 learned_addr learned my addr 172.21.15.39:0/1846551983 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.009+0000 7f31cf400640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d0110250 0x7f31d01a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7e00640 1 -- 172.21.15.39:0/1846551983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d0110250 msgr2=0x7f31d01a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7e00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d0110250 0x7f31d01a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7e00640 1 -- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d01026a0 msgr2=0x7f31d019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7e00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d01026a0 0x7f31d019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7e00640 1 -- 172.21.15.39:0/1846551983 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d01a2720 con 0x7f31d0102ff0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7e00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 0x7f31d019cd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005ab0 tx=0x7f31b8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b803f070 con 0x7f31d0102ff0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8031b60 con 0x7f31d0102ff0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d01a29b0 con 0x7f31d0102ff0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8035030 con 0x7f31d0102ff0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.010+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d01a9170 con 0x7f31d0102ff0 2024-09-21T01:00:18.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.011+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d006c110 con 0x7f31d0102ff0 2024-09-21T01:00:18.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.014+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f31b8042070 con 0x7f31d0102ff0 2024-09-21T01:00:18.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.016+0000 7f31c7400640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f31b40b6560 0x7f31b40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:18.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.016+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f31b8107de0 con 0x7f31d0102ff0 2024-09-21T01:00:18.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.016+0000 7f31cea00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f31b40b6560 0x7f31b40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:18.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.017+0000 7f31cea00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f31b40b6560 0x7f31b40b8a70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009bd0 tx=0x7f31c0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:18.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.017+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f31b80d2d80 con 0x7f31d0102ff0 2024-09-21T01:00:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:18.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:18.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: Deploying daemon mds.i on smithi160 2024-09-21T01:00:18.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6836/555411539,v1:172.21.15.150:6837/555411539] up:boot 2024-09-21T01:00:18.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: fsmap 8 up:standby 2024-09-21T01:00:18.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:17 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T01:00:18.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.260+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31d0102ab0 con 0x7f31d0102ff0 2024-09-21T01:00:18.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.261+0000 7f31c7400640 1 -- 172.21.15.39:0/1846551983 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7216 (secure 0 0 0) 0x7f31d0102ab0 con 0x7f31d0102ff0 2024-09-21T01:00:18.262 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:18.262 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":9,"btime":"2024-09-21T01:00:17:665735+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24725,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24727,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-21T01:00:18.262 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 9 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f31b40b6560 msgr2=0x7f31b40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f31b40b6560 0x7f31b40b8a70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009bd0 tx=0x7f31c0005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 msgr2=0x7f31d019cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 0x7f31d019cd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005ab0 tx=0x7f31b8003c10 comp rx=0 tx=0).stop 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 shutdown_connections 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f31b40b6560 0x7f31b40b8a70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d0110250 0x7f31d01a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f31d0102ff0 0x7f31d019cd50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 --2- 172.21.15.39:0/1846551983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31d01026a0 0x7f31d019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.263+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 >> 172.21.15.39:0/1846551983 conn(0x7f31d0079650 msgr2=0x7f31d0108cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:18.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.264+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 shutdown_connections 2024-09-21T01:00:18.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:18.264+0000 7f31d5a00640 1 -- 172.21.15.39:0/1846551983 wait complete. 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: Deploying daemon mds.i on smithi160 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6836/555411539,v1:172.21.15.150:6837/555411539] up:boot 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: fsmap 8 up:standby 2024-09-21T01:00:18.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:17 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T01:00:19.028 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:18 smithi160 ceph-mon[34696]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:19.028 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:18 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1846551983' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:19.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:18 smithi039 ceph-mon[30415]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:19.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:18 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1846551983' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:19.282 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:18 smithi150 ceph-mon[34945]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:19.282 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:18 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1846551983' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:19.607 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:00:19 smithi160 systemd[1]: Started Ceph mds.i for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:20.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: Deploying daemon mds.g on smithi039 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] up:boot 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: fsmap 9 up:standby 2024-09-21T01:00:20.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:20 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: Deploying daemon mds.g on smithi039 2024-09-21T01:00:20.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] up:boot 2024-09-21T01:00:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: fsmap 9 up:standby 2024-09-21T01:00:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:20 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T01:00:20.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: Deploying daemon mds.g on smithi039 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] up:boot 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: fsmap 9 up:standby 2024-09-21T01:00:20.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:20 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T01:00:21.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:21.508 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:21.649 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:21 smithi039 ceph-mon[30415]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:21 smithi150 ceph-mon[34945]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:21 smithi160 ceph-mon[34696]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 -- 172.21.15.39:0/1214607550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc076100 msgr2=0x7fd5dc10fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/1214607550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc076100 0x7fd5dc10fb90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009900 tx=0x7fd5c402a410 comp rx=0 tx=0).stop 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 -- 172.21.15.39:0/1214607550 shutdown_connections 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/1214607550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc1100d0 0x7fd5dc112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/1214607550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc076100 0x7fd5dc10fb90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/1214607550 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 -- 172.21.15.39:0/1214607550 >> 172.21.15.39:0/1214607550 conn(0x7fd5dc0ff5e0 msgr2=0x7fd5dc101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 -- 172.21.15.39:0/1214607550 shutdown_connections 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.432+0000 7fd5e2c00640 1 -- 172.21.15.39:0/1214607550 wait complete. 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.433+0000 7fd5e2c00640 1 Processor -- start 2024-09-21T01:00:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 -- start start 2024-09-21T01:00:22.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc076100 0x7fd5dc19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5dc1100d0 0x7fd5dc1a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc115c50 con 0x7fd5dc0757b0 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc115ad0 con 0x7fd5dc076100 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5e2c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc115dd0 con 0x7fd5dc1100d0 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39858/0 (socket says 172.21.15.39:39858) 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.434+0000 7fd5dbe00640 1 -- 172.21.15.39:0/340500853 learned_addr learned my addr 172.21.15.39:0/340500853 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5d3e00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc076100 0x7fd5dc19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5e0c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5dc1100d0 0x7fd5dc1a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5d3e00640 1 -- 172.21.15.39:0/340500853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5dc1100d0 msgr2=0x7fd5dc1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5d3e00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5dc1100d0 0x7fd5dc1a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5d3e00640 1 -- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 msgr2=0x7fd5dc19c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5d3e00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc19c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5d3e00640 1 -- 172.21.15.39:0/340500853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc1a2630 con 0x7fd5dc076100 2024-09-21T01:00:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.435+0000 7fd5dbe00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc19c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.436+0000 7fd5d3e00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc076100 0x7fd5dc19cc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402a930 tx=0x7fd5c4031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.436+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c403f070 con 0x7fd5dc076100 2024-09-21T01:00:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.436+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5c4003aa0 con 0x7fd5dc076100 2024-09-21T01:00:22.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.436+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5dc1a28c0 con 0x7fd5dc076100 2024-09-21T01:00:22.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.436+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4035070 con 0x7fd5dc076100 2024-09-21T01:00:22.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.436+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5dc1a9080 con 0x7fd5dc076100 2024-09-21T01:00:22.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.437+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5dc075bc0 con 0x7fd5dc076100 2024-09-21T01:00:22.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.440+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fd5c403bb80 con 0x7fd5dc076100 2024-09-21T01:00:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.441+0000 7fd5d9600640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5c00b65e0 0x7fd5c00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:22.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.441+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fd5c41072d0 con 0x7fd5dc076100 2024-09-21T01:00:22.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.441+0000 7fd5dbe00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5c00b65e0 0x7fd5c00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:22.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.442+0000 7fd5dbe00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5c00b65e0 0x7fd5c00b8af0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc011550 tx=0x7fd5cc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:22.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.443+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd5c40cde90 con 0x7fd5dc076100 2024-09-21T01:00:22.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.689+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5dc1a9280 con 0x7fd5dc076100 2024-09-21T01:00:22.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.690+0000 7fd5d9600640 1 -- 172.21.15.39:0/340500853 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8046 (secure 0 0 0) 0x7fd5c40d2270 con 0x7fd5dc076100 2024-09-21T01:00:22.691 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:22.691 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":10,"btime":"2024-09-21T01:00:19:843203+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24725,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24727,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24742,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T01:00:22.691 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 10 2024-09-21T01:00:22.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5c00b65e0 msgr2=0x7fd5c00b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:22.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5c00b65e0 0x7fd5c00b8af0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc011550 tx=0x7fd5cc005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc076100 msgr2=0x7fd5dc19cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc076100 0x7fd5dc19cc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402a930 tx=0x7fd5c4031a20 comp rx=0 tx=0).stop 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 shutdown_connections 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5dc1100d0 0x7fd5dc1a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5c00b65e0 0x7fd5c00b8af0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5dc076100 0x7fd5dc19cc60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 --2- 172.21.15.39:0/340500853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5dc0757b0 0x7fd5dc19c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 >> 172.21.15.39:0/340500853 conn(0x7fd5dc0ff5e0 msgr2=0x7fd5dc108b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 shutdown_connections 2024-09-21T01:00:22.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:22.692+0000 7fd5e2c00640 1 -- 172.21.15.39:0/340500853 wait complete. 2024-09-21T01:00:23.488 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:23 smithi039 ceph-mon[30415]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:23.488 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:23 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/340500853' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:23.489 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:00:23 smithi039 bash[71997]: 2b422ce1ca6abab4c85998a6ebd04093c3811da6af74c101e3ce6e90c37f0210 2024-09-21T01:00:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:23 smithi150 ceph-mon[34945]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:23 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/340500853' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:23 smithi160 ceph-mon[34696]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:23 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/340500853' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:23.903 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:00:23 smithi039 systemd[1]: Started Ceph mds.g for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:23.904 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:00:23 smithi039 ceph-mds[72032]: -- 172.21.15.39:0/59692508 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560a8ac65860 con 0x560a8ee0d600 2024-09-21T01:00:23.904 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:00:23 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g[72024]: starting mds.g at 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: Deploying daemon mds.a on smithi039 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.39:6838/443106775,v1:172.21.15.39:6839/443106775] up:boot 2024-09-21T01:00:24.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: fsmap 10 up:standby 2024-09-21T01:00:24.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: Deploying daemon mds.a on smithi039 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.39:6838/443106775,v1:172.21.15.39:6839/443106775] up:boot 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: fsmap 10 up:standby 2024-09-21T01:00:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T01:00:24.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: Deploying daemon mds.a on smithi039 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.39:6838/443106775,v1:172.21.15.39:6839/443106775] up:boot 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: fsmap 10 up:standby 2024-09-21T01:00:24.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T01:00:25.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:25 smithi160 ceph-mon[34696]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:25 smithi039 ceph-mon[30415]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:26.010 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:25 smithi150 ceph-mon[34945]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:26.555 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 systemd[1]: Starting Ceph mds.a for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:00:26.824 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 podman[72307]: 2024-09-21 01:00:26.460009097 +0000 UTC m=+0.030045995 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:00:26.824 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 podman[72307]: 2024-09-21 01:00:26.587751239 +0000 UTC m=+0.157788164 container create 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:00:26.825 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 podman[72307]: 2024-09-21 01:00:26.824429681 +0000 UTC m=+0.394466569 container init 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True) 2024-09-21T01:00:26.999 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:27.086 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 podman[72307]: 2024-09-21 01:00:26.830875013 +0000 UTC m=+0.400911902 container start 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:00:27.086 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 bash[72307]: 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c 2024-09-21T01:00:27.086 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 systemd[1]: Started Ceph mds.a for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:27.086 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 ceph-mds[72344]: -- 172.21.15.39:0/2885729670 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564a40fa7860 con 0x564a4514f180 2024-09-21T01:00:27.086 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:00:26 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a[72338]: starting mds.a at 2024-09-21T01:00:27.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 -- 172.21.15.39:0/3321165774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 msgr2=0x7f724c102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:27.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 --2- 172.21.15.39:0/3321165774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c102f40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f723c009900 tx=0x7f723c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 -- 172.21.15.39:0/3321165774 shutdown_connections 2024-09-21T01:00:27.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 --2- 172.21.15.39:0/3321165774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f724c105b00 0x7f724c10f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 --2- 172.21.15.39:0/3321165774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724c1051b0 0x7f724c1055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 --2- 172.21.15.39:0/3321165774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c102f40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 -- 172.21.15.39:0/3321165774 >> 172.21.15.39:0/3321165774 conn(0x7f724c0799a0 msgr2=0x7f724c079db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:27.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 -- 172.21.15.39:0/3321165774 shutdown_connections 2024-09-21T01:00:27.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.667+0000 7f7252a00640 1 -- 172.21.15.39:0/3321165774 wait complete. 2024-09-21T01:00:27.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.668+0000 7f7252a00640 1 Processor -- start 2024-09-21T01:00:27.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 -- start start 2024-09-21T01:00:27.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:27.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724c1051b0 0x7f724c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f724c105b00 0x7f724c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f724c112a00 con 0x7f724c102b30 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f724c112880 con 0x7f724c1051b0 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.669+0000 7f7252a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f724c112b80 con 0x7f724c105b00 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c19a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59340/0 (socket says 172.21.15.39:59340) 2024-09-21T01:00:27.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 -- 172.21.15.39:0/2447467468 learned_addr learned my addr 172.21.15.39:0/2447467468 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f7250a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f724c105b00 0x7f724c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f7243e00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724c1051b0 0x7f724c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 -- 172.21.15.39:0/2447467468 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f724c105b00 msgr2=0x7f724c19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f724c105b00 0x7f724c19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 -- 172.21.15.39:0/2447467468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724c1051b0 msgr2=0x7f724c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724c1051b0 0x7f724c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.670+0000 7f724be00640 1 -- 172.21.15.39:0/2447467468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724c1a0430 con 0x7f724c102b30 2024-09-21T01:00:27.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.671+0000 7f724be00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c19a4c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f723c005940 tx=0x7f723c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:27.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.671+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c03f030 con 0x7f724c102b30 2024-09-21T01:00:27.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.671+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724c1a06c0 con 0x7f724c102b30 2024-09-21T01:00:27.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.671+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f723c0044c0 con 0x7f724c102b30 2024-09-21T01:00:27.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.671+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c035070 con 0x7f724c102b30 2024-09-21T01:00:27.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.671+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f724c1a6e80 con 0x7f724c102b30 2024-09-21T01:00:27.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.672+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f724c103d50 con 0x7f724c102b30 2024-09-21T01:00:27.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.675+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f723c042030 con 0x7f724c102b30 2024-09-21T01:00:27.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.678+0000 7f7249600640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72300b6560 0x7f72300b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:27.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.678+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f723c1076b0 con 0x7f724c102b30 2024-09-21T01:00:27.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.678+0000 7f7243e00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72300b6560 0x7f72300b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:27.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.679+0000 7f7243e00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72300b6560 0x7f72300b8a70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f724c107f80 tx=0x7f7234005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:27.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.681+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f723c0ce270 con 0x7f724c102b30 2024-09-21T01:00:27.703 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:27.703 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.703 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.703 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.703 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:27.704 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:27.704 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:27 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:27.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:27 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:00:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:00:27.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:27 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:27.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.920+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f724c102f40 con 0x7f724c102b30 2024-09-21T01:00:27.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.921+0000 7f7249600640 1 -- 172.21.15.39:0/2447467468 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9695 (secure 0 0 0) 0x7f724c102f40 con 0x7f724c102b30 2024-09-21T01:00:27.923 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:27.923 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":12,"btime":"2024-09-21T01:00:27:589799+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24725,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24727,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24742,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-09-21T01:00:27.924 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 12 2024-09-21T01:00:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72300b6560 msgr2=0x7f72300b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72300b6560 0x7f72300b8a70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f724c107f80 tx=0x7f7234005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 msgr2=0x7f724c19a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c19a4c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f723c005940 tx=0x7f723c0040c0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 shutdown_connections 2024-09-21T01:00:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f724c105b00 0x7f724c19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72300b6560 0x7f72300b8a70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724c1051b0 0x7f724c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 --2- 172.21.15.39:0/2447467468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f724c102b30 0x7f724c19a4c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 >> 172.21.15.39:0/2447467468 conn(0x7f724c0799a0 msgr2=0x7f724c101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 shutdown_connections 2024-09-21T01:00:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:27.924+0000 7f7252a00640 1 -- 172.21.15.39:0/2447467468 wait complete. 2024-09-21T01:00:28.756 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:28 smithi150 ceph-mon[34945]: Deploying daemon mds.b on smithi150 2024-09-21T01:00:28.756 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:28 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] up:boot 2024-09-21T01:00:28.756 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:28 smithi150 ceph-mon[34945]: fsmap 11 up:standby 2024-09-21T01:00:28.756 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T01:00:28.756 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:28 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2447467468' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:28 smithi039 ceph-mon[30415]: Deploying daemon mds.b on smithi150 2024-09-21T01:00:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:28 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] up:boot 2024-09-21T01:00:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:28 smithi039 ceph-mon[30415]: fsmap 11 up:standby 2024-09-21T01:00:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T01:00:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:28 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2447467468' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:29.064 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:28 smithi150 systemd[1]: Starting Ceph mds.b for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:00:29.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:28 smithi160 ceph-mon[34696]: Deploying daemon mds.b on smithi150 2024-09-21T01:00:29.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:28 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] up:boot 2024-09-21T01:00:29.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:28 smithi160 ceph-mon[34696]: fsmap 11 up:standby 2024-09-21T01:00:29.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T01:00:29.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:28 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2447467468' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:29.446 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 podman[64437]: 2024-09-21 01:00:29.056225209 +0000 UTC m=+0.015751169 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:00:29.446 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 podman[64437]: 2024-09-21 01:00:29.185426781 +0000 UTC m=+0.144952810 container create 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:29 smithi150 ceph-mon[34945]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 podman[64437]: 2024-09-21 01:00:29.446471254 +0000 UTC m=+0.405997229 container init 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 podman[64437]: 2024-09-21 01:00:29.449519888 +0000 UTC m=+0.409045849 container start 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 ceph-mds[64456]: -- 172.21.15.150:0/1807936774 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648bdf79860 con 0x5648c2121a80 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: starting mds.b at 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 bash[64437]: 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 2024-09-21T01:00:29.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:29 smithi150 systemd[1]: Started Ceph mds.b for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:00:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:29 smithi039 ceph-mon[30415]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:29 smithi160 ceph-mon[34696]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:30.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] up:boot 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: fsmap 12 up:standby 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:30.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:30.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:30.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] up:boot 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: fsmap 12 up:standby 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:00:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] up:boot 2024-09-21T01:00:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: fsmap 12 up:standby 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:00:31.409 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:31.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:31 smithi039 ceph-mon[30415]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:31.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:31.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:31 smithi150 ceph-mon[34945]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 -- 172.21.15.39:0/3217055290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 msgr2=0x7ff1fc104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 --2- 172.21.15.39:0/3217055290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc104e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009900 tx=0x7ff1ec02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 -- 172.21.15.39:0/3217055290 shutdown_connections 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 --2- 172.21.15.39:0/3217055290 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc10de40 0x7ff1fc110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 --2- 172.21.15.39:0/3217055290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1fc105360 0x7ff1fc10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 --2- 172.21.15.39:0/3217055290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc104e20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 -- 172.21.15.39:0/3217055290 >> 172.21.15.39:0/3217055290 conn(0x7ff1fc079720 msgr2=0x7ff1fc079b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 -- 172.21.15.39:0/3217055290 shutdown_connections 2024-09-21T01:00:32.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.075+0000 7ff202e00640 1 -- 172.21.15.39:0/3217055290 wait complete. 2024-09-21T01:00:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.077+0000 7ff202e00640 1 Processor -- start 2024-09-21T01:00:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.077+0000 7ff202e00640 1 -- start start 2024-09-21T01:00:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.077+0000 7ff202e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff202e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc105360 0x7ff1fc102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff202e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1fc10de40 0x7ff1fc1031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff202e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc113aa0 con 0x7ff1fc104a10 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff202e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc113920 con 0x7ff1fc10de40 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff202e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc113c20 con 0x7ff1fc105360 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff200e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1fc10de40 0x7ff1fc1031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc102750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59366/0 (socket says 172.21.15.39:59366) 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fbe00640 1 -- 172.21.15.39:0/3405063448 learned_addr learned my addr 172.21.15.39:0/3405063448 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fb400640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc105360 0x7ff1fc102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fb400640 1 -- 172.21.15.39:0/3405063448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1fc10de40 msgr2=0x7ff1fc1031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fb400640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1fc10de40 0x7ff1fc1031d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fb400640 1 -- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 msgr2=0x7ff1fc102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.078+0000 7ff1fb400640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.079+0000 7ff1fb400640 1 -- 172.21.15.39:0/3405063448 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc1a87b0 con 0x7ff1fc105360 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.079+0000 7ff1fb400640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc105360 0x7ff1fc102c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8005fd0 tx=0x7ff1e800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.079+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8009c00 con 0x7ff1fc105360 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.079+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e8009da0 con 0x7ff1fc105360 2024-09-21T01:00:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.079+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8003e10 con 0x7ff1fc105360 2024-09-21T01:00:32.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.079+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1fc1a8aa0 con 0x7ff1fc105360 2024-09-21T01:00:32.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.080+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1fc1a8fe0 con 0x7ff1fc105360 2024-09-21T01:00:32.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.081+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1fc104e20 con 0x7ff1fc105360 2024-09-21T01:00:32.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.084+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff1e8004000 con 0x7ff1fc105360 2024-09-21T01:00:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.087+0000 7ff1f8c00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1e00b66b0 0x7ff1e00b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.087+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7ff1e80e6da0 con 0x7ff1fc105360 2024-09-21T01:00:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.087+0000 7ff1fbe00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1e00b66b0 0x7ff1e00b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:32.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.087+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff1e8017020 con 0x7ff1fc105360 2024-09-21T01:00:32.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.088+0000 7ff1fbe00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1e00b66b0 0x7ff1e00b8bc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009230 tx=0x7ff1ec0058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:31 smithi160 ceph-mon[34696]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.360+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1fc1a93b0 con 0x7ff1fc105360 2024-09-21T01:00:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.361+0000 7ff1f8c00640 1 -- 172.21.15.39:0/3405063448 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10521 (secure 0 0 0) 0x7ff1e80b1e40 con 0x7ff1fc105360 2024-09-21T01:00:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:32.363 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":13,"btime":"2024-09-21T01:00:29:614148+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24725,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24727,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24742,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-21T01:00:32.363 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 13 2024-09-21T01:00:32.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1e00b66b0 msgr2=0x7ff1e00b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:32.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1e00b66b0 0x7ff1e00b8bc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009230 tx=0x7ff1ec0058d0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc105360 msgr2=0x7ff1fc102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:32.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc105360 0x7ff1fc102c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8005fd0 tx=0x7ff1e800dde0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 shutdown_connections 2024-09-21T01:00:32.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1e00b66b0 0x7ff1e00b8bc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1fc10de40 0x7ff1fc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1fc105360 0x7ff1fc102c90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 --2- 172.21.15.39:0/3405063448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1fc104a10 0x7ff1fc102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:32.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.365+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 >> 172.21.15.39:0/3405063448 conn(0x7ff1fc079720 msgr2=0x7ff1fc1068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:32.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.366+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 shutdown_connections 2024-09-21T01:00:32.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:32.366+0000 7ff202e00640 1 -- 172.21.15.39:0/3405063448 wait complete. 2024-09-21T01:00:32.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:32 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3405063448' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:33.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:32 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3405063448' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:33.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:32 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3405063448' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:00:33.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph --format=json-pretty osd lspools 2024-09-21T01:00:33.678 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:33.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.021 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:34.047 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:00:34.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.345+0000 7fed74a00640 1 -- 172.21.15.39:0/3753179780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed701081b0 msgr2=0x7fed701085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:34.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.345+0000 7fed74a00640 1 --2- 172.21.15.39:0/3753179780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed701081b0 0x7fed701085c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009900 tx=0x7fed5c02a410 comp rx=0 tx=0).stop 2024-09-21T01:00:34.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 -- 172.21.15.39:0/3753179780 shutdown_connections 2024-09-21T01:00:34.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 --2- 172.21.15.39:0/3753179780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed70108b00 0x7fed7010f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 --2- 172.21.15.39:0/3753179780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed701081b0 0x7fed701085c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 --2- 172.21.15.39:0/3753179780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed70102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 -- 172.21.15.39:0/3753179780 >> 172.21.15.39:0/3753179780 conn(0x7fed700fe2c0 msgr2=0x7fed70100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:34.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 -- 172.21.15.39:0/3753179780 shutdown_connections 2024-09-21T01:00:34.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.346+0000 7fed74a00640 1 -- 172.21.15.39:0/3753179780 wait complete. 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 Processor -- start 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 -- start start 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed7019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed701081b0 0x7fed7019cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed70108b00 0x7fed701a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fed701127c0 con 0x7fed701081b0 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fed70112640 con 0x7fed70102870 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.347+0000 7fed74a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fed70112940 con 0x7fed70108b00 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed701081b0 0x7fed7019cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:34.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed7019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:34.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed7019c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50798/0 (socket says 172.21.15.39:50798) 2024-09-21T01:00:34.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed701081b0 0x7fed7019cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47114/0 (socket says 172.21.15.39:47114) 2024-09-21T01:00:34.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 -- 172.21.15.39:0/3800045500 learned_addr learned my addr 172.21.15.39:0/3800045500 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6fe00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed70108b00 0x7fed701a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 -- 172.21.15.39:0/3800045500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed70108b00 msgr2=0x7fed701a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed70108b00 0x7fed701a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 -- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed701081b0 msgr2=0x7fed7019cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed701081b0 0x7fed7019cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6f400640 1 -- 172.21.15.39:0/3800045500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed701a2490 con 0x7fed70102870 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.348+0000 7fed6fe00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed70108b00 0x7fed701a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.349+0000 7fed6f400640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed7019c520 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed6000e9f0 tx=0x7fed6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.349+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6000cd80 con 0x7fed70102870 2024-09-21T01:00:34.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.349+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed701a2780 con 0x7fed70102870 2024-09-21T01:00:34.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.349+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed600108e0 con 0x7fed70102870 2024-09-21T01:00:34.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.349+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60018980 con 0x7fed70102870 2024-09-21T01:00:34.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.349+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed701a8ee0 con 0x7fed70102870 2024-09-21T01:00:34.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.350+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed70102c80 con 0x7fed70102870 2024-09-21T01:00:34.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.353+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fed60010430 con 0x7fed70102870 2024-09-21T01:00:34.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.354+0000 7fed67e00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed540b6560 0x7fed540b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:34.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.354+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fed600e78a0 con 0x7fed70102870 2024-09-21T01:00:34.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.354+0000 7fed6ea00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed540b6560 0x7fed540b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:34.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.355+0000 7fed6ea00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed540b6560 0x7fed540b8a70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fed5c02aa00 tx=0x7fed5c005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:34.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.356+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fed600ae460 con 0x7fed70102870 2024-09-21T01:00:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.575+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fed7002e640 con 0x7fed70102870 2024-09-21T01:00:34.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.576+0000 7fed67e00640 1 -- 172.21.15.39:0/3800045500 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7fed600b2840 con 0x7fed70102870 2024-09-21T01:00:34.577 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:34.577 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-09-21T01:00:34.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:00:34.577 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-09-21T01:00:34.577 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-09-21T01:00:34.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:00:34.578 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed540b6560 msgr2=0x7fed540b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed540b6560 0x7fed540b8a70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fed5c02aa00 tx=0x7fed5c005a20 comp rx=0 tx=0).stop 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 msgr2=0x7fed7019c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed7019c520 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed6000e9f0 tx=0x7fed6000eeb0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 shutdown_connections 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed70108b00 0x7fed701a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed701081b0 0x7fed7019cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed540b6560 0x7fed540b8a70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 --2- 172.21.15.39:0/3800045500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed70102870 0x7fed7019c520 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.578+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 >> 172.21.15.39:0/3800045500 conn(0x7fed700fe2c0 msgr2=0x7fed70106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.579+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 shutdown_connections 2024-09-21T01:00:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:34.579+0000 7fed74a00640 1 -- 172.21.15.39:0/3800045500 wait complete. 2024-09-21T01:00:34.827 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:34 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3800045500' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T01:00:35.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:34 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3800045500' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T01:00:35.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:34 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3800045500' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T01:00:35.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:35 smithi039 ceph-mon[30415]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:35.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:35.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:35.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:35.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:35 smithi150 ceph-mon[34945]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:35 smithi160 ceph-mon[34696]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:36.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:36.268 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-21T01:00:36.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-21T01:00:36.663 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 -- 172.21.15.39:0/3015690457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc0757b0 msgr2=0x7f7cbc075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/3015690457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc075bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009900 tx=0x7f7cac02a410 comp rx=0 tx=0).stop 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 -- 172.21.15.39:0/3015690457 shutdown_connections 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/3015690457 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc10ff10 0x7f7cbc112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/3015690457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 0x7f7cbc10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/3015690457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc075bc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 -- 172.21.15.39:0/3015690457 >> 172.21.15.39:0/3015690457 conn(0x7f7cbc0ff440 msgr2=0x7f7cbc101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 -- 172.21.15.39:0/3015690457 shutdown_connections 2024-09-21T01:00:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.291+0000 7f7cbbe00640 1 -- 172.21.15.39:0/3015690457 wait complete. 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.292+0000 7f7cbbe00640 1 Processor -- start 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.292+0000 7f7cbbe00640 1 -- start start 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 0x7f7cbc19ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc10ff10 0x7f7cbc1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc115ac0 con 0x7f7cbc10ff10 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbbe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc115940 con 0x7f7cbc076100 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbbe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc115c40 con 0x7f7cbc0757b0 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cba000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 0x7f7cbc19ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc10ff10 0x7f7cbc1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbaa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc19c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:45704/0 (socket says 172.21.15.39:45704) 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.293+0000 7f7cbaa00640 1 -- 172.21.15.39:0/2293280971 learned_addr learned my addr 172.21.15.39:0/2293280971 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbaa00640 1 -- 172.21.15.39:0/2293280971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 msgr2=0x7f7cbc19ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbaa00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 0x7f7cbc19ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbaa00640 1 -- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc10ff10 msgr2=0x7f7cbc1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbaa00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc10ff10 0x7f7cbc1a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbaa00640 1 -- 172.21.15.39:0/2293280971 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc1a23e0 con 0x7f7cbc0757b0 2024-09-21T01:00:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cba000640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 0x7f7cbc19ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbaa00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc19c470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cac005940 tx=0x7f7cac0040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac03f030 con 0x7f7cbc0757b0 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cbc1a2670 con 0x7f7cbc0757b0 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cac0044c0 con 0x7f7cbc0757b0 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.294+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac035070 con 0x7f7cbc0757b0 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.295+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cbc1a8e30 con 0x7f7cbc0757b0 2024-09-21T01:00:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.296+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cbc076cc0 con 0x7f7cbc0757b0 2024-09-21T01:00:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.298+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f7cac042030 con 0x7f7cbc0757b0 2024-09-21T01:00:37.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.299+0000 7f7cb3400640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ca00b6560 0x7f7ca00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:37.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.300+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f7cac107380 con 0x7f7cbc0757b0 2024-09-21T01:00:37.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.300+0000 7f7cba000640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ca00b6560 0x7f7ca00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:37.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.300+0000 7f7cba000640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ca00b6560 0x7f7ca00b8a70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc1004e0 tx=0x7f7ca8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.304+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7cac0d2320 con 0x7f7cbc0757b0 2024-09-21T01:00:37.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:37.529+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f7cbc1a9030 con 0x7f7cbc0757b0 2024-09-21T01:00:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:00:37.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:37.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2293280971' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T01:00:37.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:37 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2293280971' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T01:00:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:37 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2293280971' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T01:00:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:37 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T01:00:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.421+0000 7f7cb3400640 1 -- 172.21.15.39:0/2293280971 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7f7cac0d7260 con 0x7f7cbc0757b0 2024-09-21T01:00:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-09-21T01:00:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ca00b6560 msgr2=0x7f7ca00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ca00b6560 0x7f7ca00b8a70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc1004e0 tx=0x7f7ca8005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 msgr2=0x7f7cbc19c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc19c470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cac005940 tx=0x7f7cac0040c0 comp rx=0 tx=0).stop 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 shutdown_connections 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cbc10ff10 0x7f7cbc1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7ca00b6560 0x7f7ca00b8a70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cbc076100 0x7f7cbc19ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 --2- 172.21.15.39:0/2293280971 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cbc0757b0 0x7f7cbc19c470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 >> 172.21.15.39:0/2293280971 conn(0x7f7cbc0ff440 msgr2=0x7f7cbc100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 shutdown_connections 2024-09-21T01:00:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:38.425+0000 7f7cbbe00640 1 -- 172.21.15.39:0/2293280971 wait complete. 2024-09-21T01:00:39.395 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-21T01:00:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:39 smithi039 ceph-mon[30415]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:39 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T01:00:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:39 smithi039 ceph-mon[30415]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T01:00:39.790 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:39 smithi150 ceph-mon[34945]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:39 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T01:00:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:39 smithi150 ceph-mon[34945]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T01:00:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:39 smithi160 ceph-mon[34696]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:39.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:39 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T01:00:39.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:39 smithi160 ceph-mon[34696]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T01:00:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 -- 172.21.15.39:0/462563817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc410c020 msgr2=0x7effc4112850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 --2- 172.21.15.39:0/462563817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc410c020 0x7effc4112850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7effbc009ca0 tx=0x7effbc02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:00:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 -- 172.21.15.39:0/462563817 shutdown_connections 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 --2- 172.21.15.39:0/462563817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc410c020 0x7effc4112850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 --2- 172.21.15.39:0/462563817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410b6d0 0x7effc410bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 --2- 172.21.15.39:0/462563817 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc4105d90 0x7effc41061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 -- 172.21.15.39:0/462563817 >> 172.21.15.39:0/462563817 conn(0x7effc4101860 msgr2=0x7effc4103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 -- 172.21.15.39:0/462563817 shutdown_connections 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.440+0000 7effc9a00640 1 -- 172.21.15.39:0/462563817 wait complete. 2024-09-21T01:00:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.441+0000 7effc9a00640 1 Processor -- start 2024-09-21T01:00:40.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 -- start start 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc4105d90 0x7effc419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 0x7effc419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 0x7effc41a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7effc4115ce0 con 0x7effc410c020 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effc4115b60 con 0x7effc4105d90 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc9a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7effc4115e60 con 0x7effc410b6d0 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.442+0000 7effc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 0x7effc41a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effbbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 0x7effc419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc2a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc4105d90 0x7effc419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 0x7effc41a1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47152/0 (socket says 172.21.15.39:47152) 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effbbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 0x7effc419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:45734/0 (socket says 172.21.15.39:45734) 2024-09-21T01:00:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 -- 172.21.15.39:0/3058008483 learned_addr learned my addr 172.21.15.39:0/3058008483 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 -- 172.21.15.39:0/3058008483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 msgr2=0x7effc419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 0x7effc419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 -- 172.21.15.39:0/3058008483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc4105d90 msgr2=0x7effc419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc4105d90 0x7effc419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 -- 172.21.15.39:0/3058008483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc41a2600 con 0x7effc410c020 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effbbe00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 0x7effc419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.443+0000 7effc3400640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 0x7effc41a1ed0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7effbc005cc0 tx=0x7effbc031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.449+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc003b70 con 0x7effc410c020 2024-09-21T01:00:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.449+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effbc0318f0 con 0x7effc410c020 2024-09-21T01:00:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.449+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc02f950 con 0x7effc410c020 2024-09-21T01:00:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.450+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc41a2890 con 0x7effc410c020 2024-09-21T01:00:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.450+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc41a9050 con 0x7effc410c020 2024-09-21T01:00:40.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.451+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc41061a0 con 0x7effc410c020 2024-09-21T01:00:40.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.455+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7effbc004b40 con 0x7effc410c020 2024-09-21T01:00:40.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.457+0000 7effbb400640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7effa80b65e0 0x7effa80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:40.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.457+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7712+0+0 (secure 0 0 0) 0x7effbc1083a0 con 0x7effc410c020 2024-09-21T01:00:40.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.457+0000 7effc2a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7effa80b65e0 0x7effa80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:40.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.458+0000 7effc2a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7effa80b65e0 0x7effa80b8af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7effb40095f0 tx=0x7effb4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:40.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.459+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7effbc0cee10 con 0x7effc410c020 2024-09-21T01:00:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:40 smithi039 ceph-mon[30415]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T01:00:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:40 smithi039 ceph-mon[30415]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T01:00:40.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:40.661+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7effc419d7c0 con 0x7effc410c020 2024-09-21T01:00:40.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:40 smithi150 ceph-mon[34945]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T01:00:40.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:40 smithi150 ceph-mon[34945]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T01:00:40.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:40 smithi160 ceph-mon[34696]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T01:00:40.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:40 smithi160 ceph-mon[34696]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T01:00:41.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.455+0000 7effbb400640 1 -- 172.21.15.39:0/3058008483 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) ==== 171+0+0 (secure 0 0 0) 0x7effc419d7c0 con 0x7effc410c020 2024-09-21T01:00:41.457 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-09-21T01:00:41.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7effa80b65e0 msgr2=0x7effa80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:41.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7effa80b65e0 0x7effa80b8af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7effb40095f0 tx=0x7effb4005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:41.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 msgr2=0x7effc41a1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:41.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 0x7effc41a1ed0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7effbc005cc0 tx=0x7effbc031700 comp rx=0 tx=0).stop 2024-09-21T01:00:41.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 shutdown_connections 2024-09-21T01:00:41.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc410c020 0x7effc41a1ed0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:41.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7effc410b6d0 0x7effc419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:41.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7effa80b65e0 0x7effa80b8af0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:41.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 --2- 172.21.15.39:0/3058008483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effc4105d90 0x7effc419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:41.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.459+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 >> 172.21.15.39:0/3058008483 conn(0x7effc4101860 msgr2=0x7effc4108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:41.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.460+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 shutdown_connections 2024-09-21T01:00:41.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:41.460+0000 7effc9a00640 1 -- 172.21.15.39:0/3058008483 wait complete. 2024-09-21T01:00:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:41 smithi150 ceph-mon[34945]: pgmap v288: 33 pgs: 4 creating+peering, 4 active+clean, 25 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:41 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3058008483' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T01:00:41.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:41 smithi160 ceph-mon[34696]: pgmap v288: 33 pgs: 4 creating+peering, 4 active+clean, 25 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:41.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:41 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3058008483' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T01:00:41.867 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:41 smithi039 ceph-mon[30415]: pgmap v288: 33 pgs: 4 creating+peering, 4 active+clean, 25 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:41.867 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:41 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3058008483' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T01:00:42.398 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-21T01:00:42.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:42 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3058008483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T01:00:42.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:42 smithi039 ceph-mon[30415]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T01:00:42.796 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:42.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:42 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3058008483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T01:00:42.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:42 smithi150 ceph-mon[34945]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T01:00:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:42 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3058008483' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T01:00:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:42 smithi160 ceph-mon[34696]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T01:00:43.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/637901970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb01094a0 msgr2=0x7f2bb01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:43.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/637901970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb01094a0 0x7f2bb01098b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009860 tx=0x7f2b9c02a340 comp rx=0 tx=0).stop 2024-09-21T01:00:43.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/637901970 shutdown_connections 2024-09-21T01:00:43.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/637901970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb0109df0 0x7f2bb0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:43.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/637901970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb01094a0 0x7f2bb01098b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:43.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/637901970 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bb0103b60 0x7f2bb0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:43.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/637901970 >> 172.21.15.39:0/637901970 conn(0x7f2bb007b850 msgr2=0x7f2bb007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:43.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.502+0000 7f2bb4e00640 1 -- 172.21.15.39:0/637901970 shutdown_connections 2024-09-21T01:00:43.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.502+0000 7f2bb4e00640 1 -- 172.21.15.39:0/637901970 wait complete. 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 Processor -- start 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 -- start start 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bb0103b60 0x7f2bb019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 0x7f2bb019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb0109df0 0x7f2bb01a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0113ab0 con 0x7f2bb01094a0 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0113930 con 0x7f2bb0109df0 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.503+0000 7f2bb4e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0113c30 con 0x7f2bb0103b60 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 0x7f2bb019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:43.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 0x7f2bb019cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58350/0 (socket says 172.21.15.39:58350) 2024-09-21T01:00:43.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 -- 172.21.15.39:0/1702245463 learned_addr learned my addr 172.21.15.39:0/1702245463 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:43.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bae800640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb0109df0 0x7f2bb01a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:43.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bade00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bb0103b60 0x7f2bb019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:43.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 -- 172.21.15.39:0/1702245463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bb0103b60 msgr2=0x7f2bb019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bb0103b60 0x7f2bb019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 -- 172.21.15.39:0/1702245463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb0109df0 msgr2=0x7f2bb01a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb0109df0 0x7f2bb01a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.504+0000 7f2bad400640 1 -- 172.21.15.39:0/1702245463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb01a25f0 con 0x7f2bb01094a0 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.505+0000 7f2bad400640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 0x7f2bb019cc20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c035040 tx=0x7f2b9c032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.505+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c037040 con 0x7f2bb01094a0 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.505+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9c030c10 con 0x7f2bb01094a0 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.505+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c0035f0 con 0x7f2bb01094a0 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.505+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb01a2880 con 0x7f2bb01094a0 2024-09-21T01:00:43.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.505+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb01a2cb0 con 0x7f2bb01094a0 2024-09-21T01:00:43.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.506+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb0103f70 con 0x7f2bb01094a0 2024-09-21T01:00:43.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.511+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2b9c043020 con 0x7f2bb01094a0 2024-09-21T01:00:43.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.512+0000 7f2ba6a00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2b940b6560 0x7f2b940b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:43.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.513+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8094+0+0 (secure 0 0 0) 0x7f2b9c106970 con 0x7f2bb01094a0 2024-09-21T01:00:43.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.513+0000 7f2bade00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2b940b6560 0x7f2b940b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:43.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.513+0000 7f2bade00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2b940b6560 0x7f2b940b8a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009bd0 tx=0x7f2ba0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:43.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.515+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2b9c04a060 con 0x7f2bb01094a0 2024-09-21T01:00:43.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:43.758+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2bb01a9230 con 0x7f2bb01094a0 2024-09-21T01:00:43.758 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:43 smithi039 ceph-mon[30415]: pgmap v291: 97 pgs: 11 creating+peering, 18 active+clean, 68 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:43.758 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:43 smithi039 ceph-mon[30415]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T01:00:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:43 smithi150 ceph-mon[34945]: pgmap v291: 97 pgs: 11 creating+peering, 18 active+clean, 68 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:43 smithi150 ceph-mon[34945]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T01:00:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:43 smithi160 ceph-mon[34696]: pgmap v291: 97 pgs: 11 creating+peering, 18 active+clean, 68 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:43 smithi160 ceph-mon[34696]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T01:00:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.498+0000 7f2ba6a00640 1 -- 172.21.15.39:0/1702245463 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f2b9c0cd240 con 0x7f2bb01094a0 2024-09-21T01:00:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 2 and data pool 3 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2b940b6560 msgr2=0x7f2b940b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2b940b6560 0x7f2b940b8a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009bd0 tx=0x7f2ba0005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 msgr2=0x7f2bb019cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 0x7f2bb019cc20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c035040 tx=0x7f2b9c032d50 comp rx=0 tx=0).stop 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 shutdown_connections 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2b940b6560 0x7f2b940b8a70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb0109df0 0x7f2bb01a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01094a0 0x7f2bb019cc20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 --2- 172.21.15.39:0/1702245463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bb0103b60 0x7f2bb019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 >> 172.21.15.39:0/1702245463 conn(0x7f2bb007b850 msgr2=0x7f2bb0105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:44.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.501+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 shutdown_connections 2024-09-21T01:00:44.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:44.502+0000 7f2bb4e00640 1 -- 172.21.15.39:0/1702245463 wait complete. 2024-09-21T01:00:44.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:44 smithi150 ceph-mon[34945]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T01:00:44.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:44 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1702245463' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T01:00:44.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:44 smithi150 ceph-mon[34945]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-21T01:00:44.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:44 smithi160 ceph-mon[34696]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T01:00:44.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:44 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1702245463' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T01:00:44.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:44 smithi160 ceph-mon[34696]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-21T01:00:44.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:44 smithi039 ceph-mon[30415]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T01:00:44.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:44 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1702245463' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T01:00:44.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:44 smithi039 ceph-mon[30415]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-21T01:00:45.289 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-21T01:00:45.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-21T01:00:45.670 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:45.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:45 smithi039 ceph-mon[30415]: pgmap v294: 97 pgs: 33 creating+peering, 35 active+clean, 29 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:45.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:45 smithi039 ceph-mon[30415]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T01:00:45.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1702245463' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T01:00:45.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:45 smithi039 ceph-mon[30415]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-21T01:00:45.809 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:45 smithi039 ceph-mon[30415]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-21T01:00:45.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:45 smithi150 ceph-mon[34945]: pgmap v294: 97 pgs: 33 creating+peering, 35 active+clean, 29 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:45 smithi150 ceph-mon[34945]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T01:00:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1702245463' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T01:00:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:45 smithi150 ceph-mon[34945]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-21T01:00:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:45 smithi150 ceph-mon[34945]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-21T01:00:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:45 smithi160 ceph-mon[34696]: pgmap v294: 97 pgs: 33 creating+peering, 35 active+clean, 29 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:45 smithi160 ceph-mon[34696]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T01:00:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1702245463' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T01:00:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:45 smithi160 ceph-mon[34696]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-21T01:00:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:45 smithi160 ceph-mon[34696]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-21T01:00:46.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 -- 172.21.15.39:0/2073415470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c41092f0 msgr2=0x7fd4c4109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2073415470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c41092f0 0x7fd4c4109700 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009900 tx=0x7fd4b002a410 comp rx=0 tx=0).stop 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 -- 172.21.15.39:0/2073415470 shutdown_connections 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2073415470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c4109c40 0x7fd4c4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2073415470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c41092f0 0x7fd4c4109700 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2073415470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c41039b0 0x7fd4c4103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 -- 172.21.15.39:0/2073415470 >> 172.21.15.39:0/2073415470 conn(0x7fd4c40ff440 msgr2=0x7fd4c4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 -- 172.21.15.39:0/2073415470 shutdown_connections 2024-09-21T01:00:46.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.384+0000 7fd4cae00640 1 -- 172.21.15.39:0/2073415470 wait complete. 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.385+0000 7fd4cae00640 1 Processor -- start 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.385+0000 7fd4cae00640 1 -- start start 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4cae00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c41039b0 0x7fd4c419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4cae00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c41092f0 0x7fd4c419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4cae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 0x7fd4c419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4cae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4113900 con 0x7fd4c4109c40 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4cae00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4113780 con 0x7fd4c41039b0 2024-09-21T01:00:46.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4cae00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4113a80 con 0x7fd4c41092f0 2024-09-21T01:00:46.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4ca400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 0x7fd4c419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:46.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4ca400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 0x7fd4c419fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58362/0 (socket says 172.21.15.39:58362) 2024-09-21T01:00:46.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4ca400640 1 -- 172.21.15.39:0/2118922734 learned_addr learned my addr 172.21.15.39:0/2118922734 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:46.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4c9000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c41092f0 0x7fd4c419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:46.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.386+0000 7fd4c9a00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c41039b0 0x7fd4c419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:46.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4ca400640 1 -- 172.21.15.39:0/2118922734 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c41092f0 msgr2=0x7fd4c419a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4ca400640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c41092f0 0x7fd4c419a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4ca400640 1 -- 172.21.15.39:0/2118922734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c41039b0 msgr2=0x7fd4c419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4ca400640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c41039b0 0x7fd4c419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4ca400640 1 -- 172.21.15.39:0/2118922734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c41a02a0 con 0x7fd4c4109c40 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4ca400640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 0x7fd4c419fb70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009fd0 tx=0x7fd4bc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc017070 con 0x7fd4c4109c40 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4bc003e20 con 0x7fd4c4109c40 2024-09-21T01:00:46.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc010460 con 0x7fd4c4109c40 2024-09-21T01:00:46.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.387+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c41a0590 con 0x7fd4c4109c40 2024-09-21T01:00:46.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.388+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c41a6cf0 con 0x7fd4c4109c40 2024-09-21T01:00:46.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.389+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c4103dc0 con 0x7fd4c4109c40 2024-09-21T01:00:46.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.391+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fd4bc003970 con 0x7fd4c4109c40 2024-09-21T01:00:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.394+0000 7fd4c2000640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4ac0b65e0 0x7fd4ac0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.394+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fd4bc0e6fe0 con 0x7fd4c4109c40 2024-09-21T01:00:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.394+0000 7fd4c9a00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4ac0b65e0 0x7fd4ac0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:46.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.395+0000 7fd4c9a00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4ac0b65e0 0x7fd4ac0b8af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009bd0 tx=0x7fd4b8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.396+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd4bc0b1cb0 con 0x7fd4c4109c40 2024-09-21T01:00:46.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:46.607+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fd4c419b460 con 0x7fd4c4109c40 2024-09-21T01:00:46.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:46 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] up:active 2024-09-21T01:00:46.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:46 smithi150 ceph-mon[34945]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:46.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:46 smithi150 ceph-mon[34945]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:46.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:46 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] up:active 2024-09-21T01:00:46.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:46 smithi160 ceph-mon[34696]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:46.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:46 smithi160 ceph-mon[34696]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:46.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:46 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] up:active 2024-09-21T01:00:46.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:46 smithi039 ceph-mon[30415]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:46.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:46 smithi039 ceph-mon[30415]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:47.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.526+0000 7fd4c2000640 1 -- 172.21.15.39:0/2118922734 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) ==== 157+0+0 (secure 0 0 0) 0x7fd4c419b460 con 0x7fd4c4109c40 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.529+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4ac0b65e0 msgr2=0x7fd4ac0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.529+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4ac0b65e0 0x7fd4ac0b8af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009bd0 tx=0x7fd4b8005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.529+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 msgr2=0x7fd4c419fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.529+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 0x7fd4c419fb70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009fd0 tx=0x7fd4bc00b260 comp rx=0 tx=0).stop 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 shutdown_connections 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c4109c40 0x7fd4c419fb70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c41092f0 0x7fd4c419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:47.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4ac0b65e0 0x7fd4ac0b8af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:47.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 --2- 172.21.15.39:0/2118922734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c41039b0 0x7fd4c419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:47.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 >> 172.21.15.39:0/2118922734 conn(0x7fd4c40ff440 msgr2=0x7fd4c4101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:47.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 shutdown_connections 2024-09-21T01:00:47.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:47.530+0000 7fd4cae00640 1 -- 172.21.15.39:0/2118922734 wait complete. 2024-09-21T01:00:47.772 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:47 smithi160 ceph-mon[34696]: pgmap v296: 97 pgs: 29 creating+peering, 68 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 925 B/s wr, 5 op/s 2024-09-21T01:00:47.772 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:47 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2118922734' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T01:00:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:47 smithi150 ceph-mon[34945]: pgmap v296: 97 pgs: 29 creating+peering, 68 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 925 B/s wr, 5 op/s 2024-09-21T01:00:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:47 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2118922734' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T01:00:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:47 smithi039 ceph-mon[30415]: pgmap v296: 97 pgs: 29 creating+peering, 68 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 925 B/s wr, 5 op/s 2024-09-21T01:00:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:47 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2118922734' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T01:00:48.422 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-21T01:00:48.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:48 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2118922734' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T01:00:48.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:48 smithi039 ceph-mon[30415]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T01:00:48.802 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:48 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2118922734' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T01:00:48.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:48 smithi150 ceph-mon[34945]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T01:00:48.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:48 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2118922734' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T01:00:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:48 smithi160 ceph-mon[34696]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T01:00:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 -- 172.21.15.39:0/1371483730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c10df10 msgr2=0x7fb03c110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 --2- 172.21.15.39:0/1371483730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c10df10 0x7fb03c110420 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb034009ca0 tx=0x7fb03402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:00:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 -- 172.21.15.39:0/1371483730 shutdown_connections 2024-09-21T01:00:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 --2- 172.21.15.39:0/1371483730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c10df10 0x7fb03c110420 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 --2- 172.21.15.39:0/1371483730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c105400 0x7fb03c10d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 --2- 172.21.15.39:0/1371483730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb03c1049d0 0x7fb03c104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 -- 172.21.15.39:0/1371483730 >> 172.21.15.39:0/1371483730 conn(0x7fb03c079720 msgr2=0x7fb03c079b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.450+0000 7fb042600640 1 -- 172.21.15.39:0/1371483730 shutdown_connections 2024-09-21T01:00:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.451+0000 7fb042600640 1 -- 172.21.15.39:0/1371483730 wait complete. 2024-09-21T01:00:49.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.452+0000 7fb042600640 1 Processor -- start 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.452+0000 7fb042600640 1 -- start start 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb042600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 0x7fb03c19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb042600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c105400 0x7fb03c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb042600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb03c10df10 0x7fb03c19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb042600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c113b90 con 0x7fb03c105400 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb042600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c113a10 con 0x7fb03c1049d0 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb042600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c113d10 con 0x7fb03c10df10 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 0x7fb03c19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 0x7fb03c19a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:53008/0 (socket says 172.21.15.39:53008) 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 -- 172.21.15.39:0/2535502415 learned_addr learned my addr 172.21.15.39:0/2535502415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb033e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c105400 0x7fb03c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03be00640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb03c10df10 0x7fb03c19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:49.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 -- 172.21.15.39:0/2535502415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb03c10df10 msgr2=0x7fb03c19fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb03c10df10 0x7fb03c19fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 -- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c105400 msgr2=0x7fb03c19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c105400 0x7fb03c19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb03b400640 1 -- 172.21.15.39:0/2535502415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03c1a0470 con 0x7fb03c1049d0 2024-09-21T01:00:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.453+0000 7fb033e00640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c105400 0x7fb03c19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.454+0000 7fb03b400640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 0x7fb03c19a500 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00d950 tx=0x7fb02c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.454+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c017030 con 0x7fb03c1049d0 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.454+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb02c004020 con 0x7fb03c1049d0 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.454+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0047a0 con 0x7fb03c1049d0 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.454+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03c1a0760 con 0x7fb03c1049d0 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.454+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb03c1a6ec0 con 0x7fb03c1049d0 2024-09-21T01:00:49.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.455+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03c06c1c0 con 0x7fb03c1049d0 2024-09-21T01:00:49.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.458+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb02c009a30 con 0x7fb03c1049d0 2024-09-21T01:00:49.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.459+0000 7fb038c00640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb0200b6560 0x7fb0200b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:49.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.460+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8492+0+0 (secure 0 0 0) 0x7fb02c0e7550 con 0x7fb03c1049d0 2024-09-21T01:00:49.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.460+0000 7fb033e00640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb0200b6560 0x7fb0200b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:49.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.460+0000 7fb033e00640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb0200b6560 0x7fb0200b8a70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb03c10fbd0 tx=0x7fb024005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.461+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb02c0adc80 con 0x7fb03c1049d0 2024-09-21T01:00:49.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:49.658+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb03c104de0 con 0x7fb03c1049d0 2024-09-21T01:00:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:49 smithi150 ceph-mon[34945]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 7 op/s 2024-09-21T01:00:49.814 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:00:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: 2024-09-21T01:00:49.629+0000 7f8f1b130640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-21T01:00:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:49 smithi160 ceph-mon[34696]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 7 op/s 2024-09-21T01:00:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:49 smithi039 ceph-mon[30415]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 7 op/s 2024-09-21T01:00:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:50 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2535502415' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T01:00:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:50 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T01:00:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:50 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2535502415' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T01:00:50.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:50 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T01:00:50.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:50 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2535502415' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T01:00:50.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:50 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T01:00:51.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.562+0000 7fb038c00640 1 -- 172.21.15.39:0/2535502415 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) ==== 228+0+0 (secure 0 0 0) 0x7fb02c0b2060 con 0x7fb03c1049d0 2024-09-21T01:00:51.564 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data_ec' created 2024-09-21T01:00:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.568+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb0200b6560 msgr2=0x7fb0200b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.568+0000 7fb042600640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb0200b6560 0x7fb0200b8a70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb03c10fbd0 tx=0x7fb024005f90 comp rx=0 tx=0).stop 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.568+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 msgr2=0x7fb03c19a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.568+0000 7fb042600640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 0x7fb03c19a500 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00d950 tx=0x7fb02c00de10 comp rx=0 tx=0).stop 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 shutdown_connections 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb03c10df10 0x7fb03c19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb03c105400 0x7fb03c19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb0200b6560 0x7fb0200b8a70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 --2- 172.21.15.39:0/2535502415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c1049d0 0x7fb03c19a500 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 >> 172.21.15.39:0/2535502415 conn(0x7fb03c079720 msgr2=0x7fb03c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.569+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 shutdown_connections 2024-09-21T01:00:51.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:51.570+0000 7fb042600640 1 -- 172.21.15.39:0/2535502415 wait complete. 2024-09-21T01:00:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:51 smithi150 ceph-mon[34945]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-09-21T01:00:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:51 smithi150 ceph-mon[34945]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T01:00:51.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:51 smithi160 ceph-mon[34696]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-09-21T01:00:51.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:51 smithi160 ceph-mon[34696]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T01:00:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:51 smithi039 ceph-mon[30415]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-09-21T01:00:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:51 smithi039 ceph-mon[30415]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T01:00:52.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-21T01:00:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:52 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T01:00:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:52 smithi160 ceph-mon[34696]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T01:00:52.858 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:52 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T01:00:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:52 smithi039 ceph-mon[30415]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T01:00:53.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:52 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T01:00:53.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:52 smithi150 ceph-mon[34945]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T01:00:53.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 -- 172.21.15.39:0/4246680814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 msgr2=0x7fb6e810bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:53.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 --2- 172.21.15.39:0/4246680814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e810bae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009900 tx=0x7fb6d402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:00:53.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 -- 172.21.15.39:0/4246680814 shutdown_connections 2024-09-21T01:00:53.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 --2- 172.21.15.39:0/4246680814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6e810c020 0x7fb6e8112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:53.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 --2- 172.21.15.39:0/4246680814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e810bae0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:53.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 --2- 172.21.15.39:0/4246680814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e81061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:53.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.613+0000 7fb6ece00640 1 -- 172.21.15.39:0/4246680814 >> 172.21.15.39:0/4246680814 conn(0x7fb6e8101860 msgr2=0x7fb6e8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:53.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.614+0000 7fb6ece00640 1 -- 172.21.15.39:0/4246680814 shutdown_connections 2024-09-21T01:00:53.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.614+0000 7fb6ece00640 1 -- 172.21.15.39:0/4246680814 wait complete. 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 Processor -- start 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 -- start start 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e807b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e8077920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6e810c020 0x7fb6e8078040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e8115d30 con 0x7fb6e810b6d0 2024-09-21T01:00:53.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e8115bb0 con 0x7fb6e8105d90 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6ece00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e8115eb0 con 0x7fb6e810c020 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6e5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e8077920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6e5e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e807b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6e5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e8077920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58806/0 (socket says 172.21.15.39:58806) 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6e5400640 1 -- 172.21.15.39:0/1454047001 learned_addr learned my addr 172.21.15.39:0/1454047001 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.615+0000 7fb6e5e00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e807b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:51666/0 (socket says 172.21.15.39:51666) 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e6800640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6e810c020 0x7fb6e8078040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5e00640 1 -- 172.21.15.39:0/1454047001 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6e810c020 msgr2=0x7fb6e8078040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:53.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5e00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6e810c020 0x7fb6e8078040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5e00640 1 -- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 msgr2=0x7fb6e8077920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5e00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e8077920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5e00640 1 -- 172.21.15.39:0/1454047001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e81acc30 con 0x7fb6e8105d90 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5400640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e8077920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6e5e00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e807b640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800e9f0 tx=0x7fb6d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.616+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d800ccc0 con 0x7fb6e8105d90 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.617+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6d80108e0 con 0x7fb6e8105d90 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.617+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e81ace60 con 0x7fb6e8105d90 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.617+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8018880 con 0x7fb6e8105d90 2024-09-21T01:00:53.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.617+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e81ad260 con 0x7fb6e8105d90 2024-09-21T01:00:53.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.618+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e81061a0 con 0x7fb6e8105d90 2024-09-21T01:00:53.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.620+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb6d8010430 con 0x7fb6e8105d90 2024-09-21T01:00:53.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.622+0000 7fb6dea00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb6cc0b6560 0x7fb6cc0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:53.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.622+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8985+0+0 (secure 0 0 0) 0x7fb6d80e7b70 con 0x7fb6e8105d90 2024-09-21T01:00:53.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.622+0000 7fb6e5400640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb6cc0b6560 0x7fb6cc0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:53.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.623+0000 7fb6e5400640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb6cc0b6560 0x7fb6cc0b8a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4006010 tx=0x7fb6d4005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:53.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.623+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb6d80b2490 con 0x7fb6e8105d90 2024-09-21T01:00:53.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:53.823+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb6e8078c20 con 0x7fb6e8105d90 2024-09-21T01:00:53.824 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:53 smithi039 ceph-mon[30415]: pgmap v302: 129 pgs: 4 creating+peering, 28 unknown, 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2024-09-21T01:00:53.824 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:53 smithi039 ceph-mon[30415]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T01:00:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:53 smithi160 ceph-mon[34696]: pgmap v302: 129 pgs: 4 creating+peering, 28 unknown, 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2024-09-21T01:00:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:53 smithi160 ceph-mon[34696]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T01:00:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:53 smithi150 ceph-mon[34945]: pgmap v302: 129 pgs: 4 creating+peering, 28 unknown, 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2024-09-21T01:00:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:53 smithi150 ceph-mon[34945]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T01:00:54.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.587+0000 7fb6dea00640 1 -- 172.21.15.39:0/1454047001 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7fb6e8078c20 con 0x7fb6e8105d90 2024-09-21T01:00:54.587 INFO:teuthology.orchestra.run.smithi039.stderr:set pool 4 allow_ec_overwrites to true 2024-09-21T01:00:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb6cc0b6560 msgr2=0x7fb6cc0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb6cc0b6560 0x7fb6cc0b8a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4006010 tx=0x7fb6d4005a20 comp rx=0 tx=0).stop 2024-09-21T01:00:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 msgr2=0x7fb6e807b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e807b640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800e9f0 tx=0x7fb6d800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 shutdown_connections 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb6e810c020 0x7fb6e8078040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6e810b6d0 0x7fb6e8077920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb6cc0b6560 0x7fb6cc0b8a70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 --2- 172.21.15.39:0/1454047001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e8105d90 0x7fb6e807b640 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 >> 172.21.15.39:0/1454047001 conn(0x7fb6e8101860 msgr2=0x7fb6e810ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 shutdown_connections 2024-09-21T01:00:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:54.590+0000 7fb6ece00640 1 -- 172.21.15.39:0/1454047001 wait complete. 2024-09-21T01:00:54.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:54 smithi160 ceph-mon[34696]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T01:00:54.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:54 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1454047001' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T01:00:54.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:54 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T01:00:54.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:54 smithi039 ceph-mon[30415]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T01:00:54.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:54 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1454047001' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T01:00:54.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:54 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T01:00:55.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:54 smithi150 ceph-mon[34945]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T01:00:55.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:54 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1454047001' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T01:00:55.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:54 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T01:00:55.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-21T01:00:55.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:55 smithi160 ceph-mon[34696]: pgmap v305: 129 pgs: 11 creating+peering, 13 unknown, 105 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:55.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:55 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T01:00:55.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:55 smithi160 ceph-mon[34696]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T01:00:55.884 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:55 smithi039 ceph-mon[30415]: pgmap v305: 129 pgs: 11 creating+peering, 13 unknown, 105 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:55 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T01:00:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:55 smithi039 ceph-mon[30415]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T01:00:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:55 smithi150 ceph-mon[34945]: pgmap v305: 129 pgs: 11 creating+peering, 13 unknown, 105 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:56.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:55 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T01:00:56.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:55 smithi150 ceph-mon[34945]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T01:00:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 -- 172.21.15.39:0/1013045777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2448105d80 msgr2=0x7f2448106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 --2- 172.21.15.39:0/1013045777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2448105d80 0x7f2448106190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2438009900 tx=0x7f243802a410 comp rx=0 tx=0).stop 2024-09-21T01:00:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 -- 172.21.15.39:0/1013045777 shutdown_connections 2024-09-21T01:00:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 --2- 172.21.15.39:0/1013045777 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f244810c010 0x7f2448112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 --2- 172.21.15.39:0/1013045777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244810bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 --2- 172.21.15.39:0/1013045777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2448105d80 0x7f2448106190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:56.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 -- 172.21.15.39:0/1013045777 >> 172.21.15.39:0/1013045777 conn(0x7f2448101810 msgr2=0x7f2448103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:56.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 -- 172.21.15.39:0/1013045777 shutdown_connections 2024-09-21T01:00:56.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.565+0000 7f244f600640 1 -- 172.21.15.39:0/1013045777 wait complete. 2024-09-21T01:00:56.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.566+0000 7f244f600640 1 Processor -- start 2024-09-21T01:00:56.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 -- start start 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2448105d80 0x7f244819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f244810c010 0x7f24481a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2448115cd0 con 0x7f244810c010 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2448115b50 con 0x7f244810b6c0 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244f600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2448115e50 con 0x7f2448105d80 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f243fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f243fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244819cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:51680/0 (socket says 172.21.15.39:51680) 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244cc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2448105d80 0x7f244819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f243fe00640 1 -- 172.21.15.39:0/2413588724 learned_addr learned my addr 172.21.15.39:0/2413588724 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.567+0000 7f244d600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f244810c010 0x7f24481a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f243fe00640 1 -- 172.21.15.39:0/2413588724 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2448105d80 msgr2=0x7f244819c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f243fe00640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2448105d80 0x7f244819c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f243fe00640 1 -- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f244810c010 msgr2=0x7f24481a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f243fe00640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f244810c010 0x7f24481a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f243fe00640 1 -- 172.21.15.39:0/2413588724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24481a2630 con 0x7f244810b6c0 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f243fe00640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244819cc60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f243000e9f0 tx=0x7f243000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243000ccc0 con 0x7f244810b6c0 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24481a2920 con 0x7f244810b6c0 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24300108e0 con 0x7f244810b6c0 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430018860 con 0x7f244810b6c0 2024-09-21T01:00:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.568+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24481a9080 con 0x7f244810b6c0 2024-09-21T01:00:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.569+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2448106190 con 0x7f244810b6c0 2024-09-21T01:00:56.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.572+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2430010430 con 0x7f244810b6c0 2024-09-21T01:00:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.574+0000 7f2446000640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f242c0b6560 0x7f242c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.574+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8985+0+0 (secure 0 0 0) 0x7f24300e7de0 con 0x7f244810b6c0 2024-09-21T01:00:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.574+0000 7f244cc00640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f242c0b6560 0x7f242c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:56.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.575+0000 7f244cc00640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f242c0b6560 0x7f242c0b8a70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2438031440 tx=0x7f2438009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:56.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.576+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f24300b2700 con 0x7f244810b6c0 2024-09-21T01:00:56.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:56.820+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f244819d7f0 con 0x7f244810b6c0 2024-09-21T01:00:57.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.596+0000 7f2446000640 1 -- 172.21.15.39:0/2413588724 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f24300b7640 con 0x7f244810b6c0 2024-09-21T01:00:57.596 INFO:teuthology.orchestra.run.smithi039.stderr:added data pool 4 to fsmap 2024-09-21T01:00:57.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f242c0b6560 msgr2=0x7f242c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:57.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f242c0b6560 0x7f242c0b8a70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2438031440 tx=0x7f2438009560 comp rx=0 tx=0).stop 2024-09-21T01:00:57.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 msgr2=0x7f244819cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:57.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244819cc60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f243000e9f0 tx=0x7f243000eeb0 comp rx=0 tx=0).stop 2024-09-21T01:00:57.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 shutdown_connections 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f244810c010 0x7f24481a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f242c0b6560 0x7f242c0b8a70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244810b6c0 0x7f244819cc60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 --2- 172.21.15.39:0/2413588724 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2448105d80 0x7f244819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 >> 172.21.15.39:0/2413588724 conn(0x7f2448101810 msgr2=0x7f2448103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 shutdown_connections 2024-09-21T01:00:57.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:57.600+0000 7f244f600640 1 -- 172.21.15.39:0/2413588724 wait complete. 2024-09-21T01:00:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:57 smithi160 ceph-mon[34696]: pgmap v307: 129 pgs: 11 creating+peering, 7 unknown, 111 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:57 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2413588724' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T01:00:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:57 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T01:00:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:57 smithi039 ceph-mon[30415]: pgmap v307: 129 pgs: 11 creating+peering, 7 unknown, 111 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:57 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2413588724' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T01:00:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:57 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T01:00:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:57 smithi150 ceph-mon[34945]: pgmap v307: 129 pgs: 11 creating+peering, 7 unknown, 111 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:57 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2413588724' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T01:00:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:57 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T01:00:58.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:00:58.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:58 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T01:00:58.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:58 smithi160 ceph-mon[34696]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T01:00:58.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:58 smithi160 ceph-mon[34696]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:58.888 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:00:58.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:58 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T01:00:58.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:58 smithi039 ceph-mon[30415]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T01:00:58.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:58 smithi039 ceph-mon[30415]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:58 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T01:00:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:58 smithi150 ceph-mon[34945]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T01:00:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:58 smithi150 ceph-mon[34945]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1920153803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc1094a0 msgr2=0x7fc9bc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1920153803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc1094a0 0x7fc9bc1098b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009900 tx=0x7fc9a402a410 comp rx=0 tx=0).stop 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1920153803 shutdown_connections 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1920153803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9bc109df0 0x7fc9bc110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1920153803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc1094a0 0x7fc9bc1098b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1920153803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9bc103b60 0x7fc9bc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1920153803 >> 172.21.15.39:0/1920153803 conn(0x7fc9bc07b850 msgr2=0x7fc9bc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1920153803 shutdown_connections 2024-09-21T01:00:59.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.564+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1920153803 wait complete. 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.565+0000 7fc9c0e00640 1 Processor -- start 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.565+0000 7fc9c0e00640 1 -- start start 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9c0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 0x7fc9bc19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9c0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9bc1094a0 0x7fc9bc19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9c0e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9bc109df0 0x7fc9bc19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9c0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc113ab0 con 0x7fc9bc103b60 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9c0e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc113930 con 0x7fc9bc109df0 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9c0e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc113c30 con 0x7fc9bc1094a0 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9b9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 0x7fc9bc19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9b9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 0x7fc9bc19a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58854/0 (socket says 172.21.15.39:58854) 2024-09-21T01:00:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9b9e00640 1 -- 172.21.15.39:0/1791381864 learned_addr learned my addr 172.21.15.39:0/1791381864 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9b3e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9bc1094a0 0x7fc9bc19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.566+0000 7fc9ba800640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9bc109df0 0x7fc9bc19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b9e00640 1 -- 172.21.15.39:0/1791381864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9bc1094a0 msgr2=0x7fc9bc19aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b9e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9bc1094a0 0x7fc9bc19aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b9e00640 1 -- 172.21.15.39:0/1791381864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9bc109df0 msgr2=0x7fc9bc19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b9e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9bc109df0 0x7fc9bc19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b9e00640 1 -- 172.21.15.39:0/1791381864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc1a0410 con 0x7fc9bc103b60 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b9e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 0x7fc9bc19a4a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00e9f0 tx=0x7fc9ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac00cd80 con 0x7fc9bc103b60 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9ac0054d0 con 0x7fc9bc103b60 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac010690 con 0x7fc9bc103b60 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.567+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9bc1a0700 con 0x7fc9bc103b60 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.568+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9bc1a6e60 con 0x7fc9bc103b60 2024-09-21T01:00:59.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.569+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9bc103f70 con 0x7fc9bc103b60 2024-09-21T01:00:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.571+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fc9ac004e30 con 0x7fc9bc103b60 2024-09-21T01:00:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.572+0000 7fc9b2a00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9a00b65e0 0x7fc9a00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:00:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.573+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9017+0+0 (secure 0 0 0) 0x7fc9ac0e7c10 con 0x7fc9bc103b60 2024-09-21T01:00:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.573+0000 7fc9b3e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9a00b65e0 0x7fc9a00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:00:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.573+0000 7fc9b3e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9a00b65e0 0x7fc9a00b8af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009230 tx=0x7fc9a4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:00:59.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.575+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc9ac0ae130 con 0x7fc9bc103b60 2024-09-21T01:00:59.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:00:59 smithi039 ceph-mon[30415]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.812+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9bc19b5d0 con 0x7fc9bc103b60 2024-09-21T01:00:59.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.812+0000 7fc9b2a00640 1 -- 172.21.15.39:0/1791381864 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11839 (secure 0 0 0) 0x7fc9bc19b5d0 con 0x7fc9bc103b60 2024-09-21T01:00:59.814 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:00:59.814 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":17,"btime":"2024-09-21T01:00:57:585380+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24725,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24727,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24742,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:00:56.822076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24764,"qdb_cluster":[24764]},"id":1}]} 2024-09-21T01:00:59.814 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 17 2024-09-21T01:00:59.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:00:59 smithi150 ceph-mon[34945]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:00:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9a00b65e0 msgr2=0x7fc9a00b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9a00b65e0 0x7fc9a00b8af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009230 tx=0x7fc9a4031400 comp rx=0 tx=0).stop 2024-09-21T01:00:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 msgr2=0x7fc9bc19a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:00:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 0x7fc9bc19a4a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00e9f0 tx=0x7fc9ac00edf0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 shutdown_connections 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9a00b65e0 0x7fc9a00b8af0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9bc109df0 0x7fc9bc19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9bc1094a0 0x7fc9bc19aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1791381864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc103b60 0x7fc9bc19a4a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.815+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 >> 172.21.15.39:0/1791381864 conn(0x7fc9bc07b850 msgr2=0x7fc9bc107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:00:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.816+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 shutdown_connections 2024-09-21T01:00:59.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:00:59.816+0000 7fc9c0e00640 1 -- 172.21.15.39:0/1791381864 wait complete. 2024-09-21T01:01:00.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:00:59 smithi160 ceph-mon[34696]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:00.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:01:00.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:00 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1791381864' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:00.960 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:01.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:00 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1791381864' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:01.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:00 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1791381864' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:01.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.634+0000 7fb306e00640 1 -- 172.21.15.39:0/3181225475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3001039b0 msgr2=0x7fb300103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:01.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.634+0000 7fb306e00640 1 --2- 172.21.15.39:0/3181225475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3001039b0 0x7fb300103dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009900 tx=0x7fb2f402a410 comp rx=0 tx=0).stop 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 -- 172.21.15.39:0/3181225475 shutdown_connections 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 --2- 172.21.15.39:0/3181225475 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb300109c40 0x7fb300110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 --2- 172.21.15.39:0/3181225475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001092f0 0x7fb300109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 --2- 172.21.15.39:0/3181225475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3001039b0 0x7fb300103dc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 -- 172.21.15.39:0/3181225475 >> 172.21.15.39:0/3181225475 conn(0x7fb3000ff440 msgr2=0x7fb300101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 -- 172.21.15.39:0/3181225475 shutdown_connections 2024-09-21T01:01:01.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.635+0000 7fb306e00640 1 -- 172.21.15.39:0/3181225475 wait complete. 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.636+0000 7fb306e00640 1 Processor -- start 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.636+0000 7fb306e00640 1 -- start start 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb3001039b0 0x7fb30019c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001092f0 0x7fb30019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 0x7fb3001a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb300113950 con 0x7fb300109c40 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001137d0 con 0x7fb3001092f0 2024-09-21T01:01:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb300113ad0 con 0x7fb3001039b0 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 0x7fb3001a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb305a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb3001039b0 0x7fb30019c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 0x7fb3001a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58876/0 (socket says 172.21.15.39:58876) 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb306400640 1 -- 172.21.15.39:0/4177196631 learned_addr learned my addr 172.21.15.39:0/4177196631 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.637+0000 7fb305000640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001092f0 0x7fb30019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306400640 1 -- 172.21.15.39:0/4177196631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb3001039b0 msgr2=0x7fb30019c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:01.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306400640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb3001039b0 0x7fb30019c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306400640 1 -- 172.21.15.39:0/4177196631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001092f0 msgr2=0x7fb30019cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306400640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001092f0 0x7fb30019cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306400640 1 -- 172.21.15.39:0/4177196631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3001a2510 con 0x7fb300109c40 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306400640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 0x7fb3001a1de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800eb10 tx=0x7fb2f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8004b20 con 0x7fb300109c40 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3001a2800 con 0x7fb300109c40 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2f8004cc0 con 0x7fb300109c40 2024-09-21T01:01:01.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.638+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8010690 con 0x7fb300109c40 2024-09-21T01:01:01.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.639+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3001a8f60 con 0x7fb300109c40 2024-09-21T01:01:01.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.640+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb300103dc0 con 0x7fb300109c40 2024-09-21T01:01:01.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.642+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb2f8005000 con 0x7fb300109c40 2024-09-21T01:01:01.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.644+0000 7fb2fe000640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2e80b6560 0x7fb2e80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:01.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.644+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9017+0+0 (secure 0 0 0) 0x7fb2f80e7d60 con 0x7fb300109c40 2024-09-21T01:01:01.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.644+0000 7fb305a00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2e80b6560 0x7fb2e80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:01.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.645+0000 7fb305a00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2e80b6560 0x7fb2e80b8a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4005ab0 tx=0x7fb2f4005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:01.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.650+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb2f8056710 con 0x7fb300109c40 2024-09-21T01:01:01.742 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:01 smithi039 ceph-mon[30415]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:01 smithi150 ceph-mon[34945]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:01 smithi160 ceph-mon[34696]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.859+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb30019d6d0 con 0x7fb300109c40 2024-09-21T01:01:01.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.861+0000 7fb2fe000640 1 -- 172.21.15.39:0/4177196631 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24562 (secure 0 0 0) 0x7fb30019d6d0 con 0x7fb300109c40 2024-09-21T01:01:01.863 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:01.863 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":86,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:00:56.821922+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:01:01.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2e80b6560 msgr2=0x7fb2e80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:01.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2e80b6560 0x7fb2e80b8a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4005ab0 tx=0x7fb2f4005a20 comp rx=0 tx=0).stop 2024-09-21T01:01:01.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 msgr2=0x7fb3001a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:01.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 0x7fb3001a1de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800eb10 tx=0x7fb2f800c2d0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 shutdown_connections 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb300109c40 0x7fb3001a1de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2e80b6560 0x7fb2e80b8a70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001092f0 0x7fb30019cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 --2- 172.21.15.39:0/4177196631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb3001039b0 0x7fb30019c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 >> 172.21.15.39:0/4177196631 conn(0x7fb3000ff440 msgr2=0x7fb3001089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 shutdown_connections 2024-09-21T01:01:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:01.864+0000 7fb306e00640 1 -- 172.21.15.39:0/4177196631 wait complete. 2024-09-21T01:01:02.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:01:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:02 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4177196631' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:03.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:02 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4177196631' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:03.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:02 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4177196631' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:03.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 -- 172.21.15.39:0/3112715777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 msgr2=0x7fb860077d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 --2- 172.21.15.39:0/3112715777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb860077d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb850009900 tx=0x7fb85002a410 comp rx=0 tx=0).stop 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 -- 172.21.15.39:0/3112715777 shutdown_connections 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 --2- 172.21.15.39:0/3112715777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb860114570 0x7fb860116990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 --2- 172.21.15.39:0/3112715777 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb860078290 0x7fb860114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 --2- 172.21.15.39:0/3112715777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb860077d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 -- 172.21.15.39:0/3112715777 >> 172.21.15.39:0/3112715777 conn(0x7fb860101680 msgr2=0x7fb860103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 -- 172.21.15.39:0/3112715777 shutdown_connections 2024-09-21T01:01:03.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.748+0000 7fb865000640 1 -- 172.21.15.39:0/3112715777 wait complete. 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.749+0000 7fb865000640 1 Processor -- start 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.749+0000 7fb865000640 1 -- start start 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb865000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb8601a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb865000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb860078290 0x7fb8601a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb865000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb860114570 0x7fb8601a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb865000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb860119f40 con 0x7fb860077940 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb865000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb860119dc0 con 0x7fb860078290 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb865000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb86011a0c0 con 0x7fb860114570 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb8601a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb8601a0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52812/0 (socket says 172.21.15.39:52812) 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85f400640 1 -- 172.21.15.39:0/2823580716 learned_addr learned my addr 172.21.15.39:0/2823580716 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:03.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85ea00640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb860078290 0x7fb8601a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85fe00640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb860114570 0x7fb8601a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85f400640 1 -- 172.21.15.39:0/2823580716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb860114570 msgr2=0x7fb8601a61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.750+0000 7fb85f400640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb860114570 0x7fb8601a61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb85f400640 1 -- 172.21.15.39:0/2823580716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb860078290 msgr2=0x7fb8601a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb85f400640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb860078290 0x7fb8601a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb85f400640 1 -- 172.21.15.39:0/2823580716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8601a6900 con 0x7fb860077940 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb85f400640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb8601a0990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb850005ab0 tx=0x7fb850003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85003f070 con 0x7fb860077940 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb850031b60 con 0x7fb860077940 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8601a6b90 con 0x7fb860077940 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850035030 con 0x7fb860077940 2024-09-21T01:01:03.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.751+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8601ad350 con 0x7fb860077940 2024-09-21T01:01:03.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.752+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb860077d50 con 0x7fb860077940 2024-09-21T01:01:03.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.755+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb850042070 con 0x7fb860077940 2024-09-21T01:01:03.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.757+0000 7fb857e00640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb8440b65e0 0x7fb8440b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:03.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.758+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9017+0+0 (secure 0 0 0) 0x7fb850108500 con 0x7fb860077940 2024-09-21T01:01:03.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.758+0000 7fb85ea00640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb8440b65e0 0x7fb8440b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:03.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.758+0000 7fb85ea00640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb8440b65e0 0x7fb8440b8af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb86007a0c0 tx=0x7fb84c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:03.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.760+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb8500d2e00 con 0x7fb860077940 2024-09-21T01:01:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:03 smithi039 ceph-mon[30415]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.964+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb86010f760 con 0x7fb860077940 2024-09-21T01:01:03.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.966+0000 7fb857e00640 1 -- 172.21.15.39:0/2823580716 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24562 (secure 0 0 0) 0x7fb8500d7d40 con 0x7fb860077940 2024-09-21T01:01:03.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:03.967 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":86,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:00:56.821922+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.968+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb8440b65e0 msgr2=0x7fb8440b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.968+0000 7fb865000640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb8440b65e0 0x7fb8440b8af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb86007a0c0 tx=0x7fb84c005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.968+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 msgr2=0x7fb8601a0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.968+0000 7fb865000640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb8601a0990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb850005ab0 tx=0x7fb850003c10 comp rx=0 tx=0).stop 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 shutdown_connections 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb860114570 0x7fb8601a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb8440b65e0 0x7fb8440b8af0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb860078290 0x7fb8601a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 --2- 172.21.15.39:0/2823580716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb860077940 0x7fb8601a0990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 >> 172.21.15.39:0/2823580716 conn(0x7fb860101680 msgr2=0x7fb8601165e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 shutdown_connections 2024-09-21T01:01:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:03.969+0000 7fb865000640 1 -- 172.21.15.39:0/2823580716 wait complete. 2024-09-21T01:01:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:03 smithi150 ceph-mon[34945]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:03 smithi160 ceph-mon[34696]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:04.744 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:01:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:04 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2823580716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:05.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:04 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2823580716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:05.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:04 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2823580716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:05.121 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:05.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 -- 172.21.15.39:0/779494326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4076100 msgr2=0x7f94e410fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:05.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 --2- 172.21.15.39:0/779494326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4076100 0x7f94e410fba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009900 tx=0x7f94d002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 -- 172.21.15.39:0/779494326 shutdown_connections 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 --2- 172.21.15.39:0/779494326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94e41100e0 0x7f94e4112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 --2- 172.21.15.39:0/779494326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4076100 0x7f94e410fba0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 --2- 172.21.15.39:0/779494326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.738+0000 7f94eac00640 1 -- 172.21.15.39:0/779494326 >> 172.21.15.39:0/779494326 conn(0x7f94e40ff610 msgr2=0x7f94e4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.739+0000 7f94eac00640 1 -- 172.21.15.39:0/779494326 shutdown_connections 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.739+0000 7f94eac00640 1 -- 172.21.15.39:0/779494326 wait complete. 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.740+0000 7f94eac00640 1 Processor -- start 2024-09-21T01:01:05.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.740+0000 7f94eac00640 1 -- start start 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94eac00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94eac00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94e4076100 0x7f94e419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94eac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 0x7f94e41a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94eac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4115ce0 con 0x7f94e41100e0 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94eac00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4115b60 con 0x7f94e40757b0 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94eac00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4115e60 con 0x7f94e4076100 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 0x7f94e41a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39464/0 (socket says 172.21.15.39:39464) 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 0x7f94e41a1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52826/0 (socket says 172.21.15.39:52826) 2024-09-21T01:01:05.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e3e00640 1 -- 172.21.15.39:0/3319971866 learned_addr learned my addr 172.21.15.39:0/3319971866 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e3400640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94e4076100 0x7f94e419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e3e00640 1 -- 172.21.15.39:0/3319971866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94e4076100 msgr2=0x7f94e419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.741+0000 7f94e3e00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94e4076100 0x7f94e419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e3e00640 1 -- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 msgr2=0x7f94e41a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e3e00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 0x7f94e41a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e3e00640 1 -- 172.21.15.39:0/3319971866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e41a25e0 con 0x7f94e40757b0 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e8c00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 0x7f94e41a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:05.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e3e00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e419c670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9f0 tx=0x7f94d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:05.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d400cd80 con 0x7f94e40757b0 2024-09-21T01:01:05.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e41a28d0 con 0x7f94e40757b0 2024-09-21T01:01:05.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d40108e0 con 0x7f94e40757b0 2024-09-21T01:01:05.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4018980 con 0x7f94e40757b0 2024-09-21T01:01:05.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.742+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e41a9030 con 0x7f94e40757b0 2024-09-21T01:01:05.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.743+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e4076cc0 con 0x7f94e40757b0 2024-09-21T01:01:05.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.746+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f94d4010430 con 0x7f94e40757b0 2024-09-21T01:01:05.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.747+0000 7f94e0c00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94c80b6560 0x7f94c80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:05.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.747+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9017+0+0 (secure 0 0 0) 0x7f94d40e7e90 con 0x7f94e40757b0 2024-09-21T01:01:05.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.747+0000 7f94e3400640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94c80b6560 0x7f94c80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:05.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.748+0000 7f94e3400640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94c80b6560 0x7f94c80b8a70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f94d0031440 tx=0x7f94d0005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:05.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.749+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f94d40b2790 con 0x7f94e40757b0 2024-09-21T01:01:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:05 smithi039 ceph-mon[30415]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:05.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.987+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94e419d7a0 con 0x7f94e40757b0 2024-09-21T01:01:05.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.988+0000 7f94e0c00640 1 -- 172.21.15.39:0/3319971866 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11839 (secure 0 0 0) 0x7f94d40b76d0 con 0x7f94e40757b0 2024-09-21T01:01:05.990 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:05.990 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":17,"btime":"2024-09-21T01:00:57:585380+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24715,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24725,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24727,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24742,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:00:56.822076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24764,"qdb_cluster":[24764]},"id":1}]} 2024-09-21T01:01:05.990 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 17 2024-09-21T01:01:05.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94c80b6560 msgr2=0x7f94c80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:05.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94c80b6560 0x7f94c80b8a70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f94d0031440 tx=0x7f94d0005a20 comp rx=0 tx=0).stop 2024-09-21T01:01:05.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 msgr2=0x7f94e419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:05.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e419c670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9f0 tx=0x7f94d400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 shutdown_connections 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e41100e0 0x7f94e41a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94e4076100 0x7f94e419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94c80b6560 0x7f94c80b8a70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 --2- 172.21.15.39:0/3319971866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40757b0 0x7f94e419c670 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 >> 172.21.15.39:0/3319971866 conn(0x7f94e40ff610 msgr2=0x7f94e4109bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 shutdown_connections 2024-09-21T01:01:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:05.991+0000 7f94eac00640 1 -- 172.21.15.39:0/3319971866 wait complete. 2024-09-21T01:01:06.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:05 smithi150 ceph-mon[34945]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:05 smithi160 ceph-mon[34696]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:06.739 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:01:07.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:06 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3319971866' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:07.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:06 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3319971866' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:07.119 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:07.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:06 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3319971866' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:07.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.757+0000 7fa9d2600640 1 -- 172.21.15.39:0/1651901430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc105bb0 msgr2=0x7fa9cc105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:07.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.757+0000 7fa9d2600640 1 --2- 172.21.15.39:0/1651901430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc105fc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009900 tx=0x7fa9c002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 -- 172.21.15.39:0/1651901430 shutdown_connections 2024-09-21T01:01:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 --2- 172.21.15.39:0/1651901430 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc10be40 0x7fa9cc112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 --2- 172.21.15.39:0/1651901430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 --2- 172.21.15.39:0/1651901430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc105fc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 -- 172.21.15.39:0/1651901430 >> 172.21.15.39:0/1651901430 conn(0x7fa9cc101680 msgr2=0x7fa9cc103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 -- 172.21.15.39:0/1651901430 shutdown_connections 2024-09-21T01:01:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.758+0000 7fa9d2600640 1 -- 172.21.15.39:0/1651901430 wait complete. 2024-09-21T01:01:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.759+0000 7fa9d2600640 1 Processor -- start 2024-09-21T01:01:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.759+0000 7fa9d2600640 1 -- start start 2024-09-21T01:01:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d2600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d2600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d2600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9cc10be40 0x7fa9cc1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d2600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc115b50 con 0x7fa9cc10b4f0 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d2600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1159d0 con 0x7fa9cc10be40 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d2600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc115cd0 con 0x7fa9cc105bb0 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d1200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52842/0 (socket says 172.21.15.39:52842) 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9cbe00640 1 -- 172.21.15.39:0/4161667805 learned_addr learned my addr 172.21.15.39:0/4161667805 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:07.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.760+0000 7fa9d1c00640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9cc10be40 0x7fa9cc1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9d1200640 1 -- 172.21.15.39:0/4161667805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9cc10be40 msgr2=0x7fa9cc1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9d1200640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9cc10be40 0x7fa9cc1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9d1200640 1 -- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 msgr2=0x7fa9cc19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9d1200640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9d1200640 1 -- 172.21.15.39:0/4161667805 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9cc1a2590 con 0x7fa9cc105bb0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9cbe00640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9d1200640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc19c620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa9c002a9e0 tx=0x7fa9c0003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.761+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c003f070 con 0x7fa9cc105bb0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.762+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9c0031d60 con 0x7fa9cc105bb0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.762+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9cc1a2820 con 0x7fa9cc105bb0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.762+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0035030 con 0x7fa9cc105bb0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.762+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9cc1a8fe0 con 0x7fa9cc105bb0 2024-09-21T01:01:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.763+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9cc106e30 con 0x7fa9cc105bb0 2024-09-21T01:01:07.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.765+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fa9c0042070 con 0x7fa9cc105bb0 2024-09-21T01:01:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.766+0000 7fa9c9600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9b40b65e0 0x7fa9b40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:07.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.767+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9017+0+0 (secure 0 0 0) 0x7fa9c01082d0 con 0x7fa9cc105bb0 2024-09-21T01:01:07.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.767+0000 7fa9cbe00640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9b40b65e0 0x7fa9b40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:07.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.767+0000 7fa9cbe00640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9b40b65e0 0x7fa9b40b8af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc104d80 tx=0x7fa9b800b440 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:07.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.768+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa9c00d2bd0 con 0x7fa9cc105bb0 2024-09-21T01:01:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:07 smithi039 ceph-mon[30415]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.971+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9cc10dc40 con 0x7fa9cc105bb0 2024-09-21T01:01:07.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.973+0000 7fa9c9600640 1 -- 172.21.15.39:0/4161667805 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24562 (secure 0 0 0) 0x7fa9c00d7b10 con 0x7fa9cc105bb0 2024-09-21T01:01:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:07.974 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":86,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:00:56.821922+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.975+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9b40b65e0 msgr2=0x7fa9b40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.975+0000 7fa9d2600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9b40b65e0 0x7fa9b40b8af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc104d80 tx=0x7fa9b800b440 comp rx=0 tx=0).stop 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.975+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc105bb0 msgr2=0x7fa9cc19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.975+0000 7fa9d2600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc19c620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa9c002a9e0 tx=0x7fa9c0003cd0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 shutdown_connections 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa9b40b65e0 0x7fa9b40b8af0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9cc10be40 0x7fa9cc1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9cc10b4f0 0x7fa9cc19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 --2- 172.21.15.39:0/4161667805 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa9cc105bb0 0x7fa9cc19c620 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 >> 172.21.15.39:0/4161667805 conn(0x7fa9cc101680 msgr2=0x7fa9cc10ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 shutdown_connections 2024-09-21T01:01:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:07.976+0000 7fa9d2600640 1 -- 172.21.15.39:0/4161667805 wait complete. 2024-09-21T01:01:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:07 smithi150 ceph-mon[34945]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:07 smithi160 ceph-mon[34696]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:08.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-21T01:00:43.758917+0000', 'modified': '2024-09-21T01:00:56.822076+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24764}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24764': {'gid': 24764, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.150:6839/103174700', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 103174700}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 103174700}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24764, 'qdb_cluster': [24764]} 2024-09-21T01:01:08.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-21T01:01:08.889 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]} : dispatch 2024-09-21T01:01:08.938 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]} : dispatch 2024-09-21T01:01:08.939 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:08 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4161667805' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]} : dispatch 2024-09-21T01:01:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]} : dispatch 2024-09-21T01:01:09.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]} : dispatch 2024-09-21T01:01:09.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]} : dispatch 2024-09-21T01:01:09.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]} : dispatch 2024-09-21T01:01:09.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:08 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4161667805' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]} : dispatch 2024-09-21T01:01:09.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:08 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4161667805' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:09.265 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 -- 172.21.15.39:0/1953881155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 msgr2=0x7f923c110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 --2- 172.21.15.39:0/1953881155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 0x7f923c110670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9234009be0 tx=0x7f923402a6c0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 -- 172.21.15.39:0/1953881155 shutdown_connections 2024-09-21T01:01:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 --2- 172.21.15.39:0/1953881155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 0x7f923c110670 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 --2- 172.21.15.39:0/1953881155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c1062d0 0x7f923c1066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 --2- 172.21.15.39:0/1953881155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f923c06bd50 0x7f923c06c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 -- 172.21.15.39:0/1953881155 >> 172.21.15.39:0/1953881155 conn(0x7f923c0799a0 msgr2=0x7f923c079db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 -- 172.21.15.39:0/1953881155 shutdown_connections 2024-09-21T01:01:09.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.950+0000 7f9242000640 1 -- 172.21.15.39:0/1953881155 wait complete. 2024-09-21T01:01:09.953 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:09.953 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-09-21T01:01:09.953 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-09-21T01:01:09.953 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-09-21T01:01:09.953 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-09-21T01:01:09.953 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-21T01:01:09.954 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-09-21T01:01:09.954 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-09-21T01:01:09.954 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-21T01:01:09.954 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-09-21T01:01:09.954 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-21T01:01:09.954 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:09 smithi039 ceph-mon[30415]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T01:01:09.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.951+0000 7f9242000640 1 Processor -- start 2024-09-21T01:01:09.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 -- start start 2024-09-21T01:01:09.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 0x7f923c1043d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f923c1062d0 0x7f923c1029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 0x7f923c102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f923c113ac0 con 0x7f923c1062d0 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f923c113940 con 0x7f923c06bd50 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9242000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f923c113c40 con 0x7f923c106c20 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f923b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 0x7f923c1043d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f923b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 0x7f923c1043d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39496/0 (socket says 172.21.15.39:39496) 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f923b400640 1 -- 172.21.15.39:0/736415344 learned_addr learned my addr 172.21.15.39:0/736415344 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.952+0000 7f9233e00640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f923c1062d0 0x7f923c1029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923be00640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 0x7f923c102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923b400640 1 -- 172.21.15.39:0/736415344 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 msgr2=0x7f923c102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923b400640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 0x7f923c102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923b400640 1 -- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f923c1062d0 msgr2=0x7f923c1029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923b400640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f923c1062d0 0x7f923c1029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923b400640 1 -- 172.21.15.39:0/736415344 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c103720 con 0x7f923c06bd50 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.953+0000 7f923b400640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 0x7f923c1043d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f922c00e9c0 tx=0x7f922c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.954+0000 7f9238c00640 1 -- 172.21.15.39:0/736415344 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c00cde0 con 0x7f923c06bd50 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.954+0000 7f9238c00640 1 -- 172.21.15.39:0/736415344 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f922c0035f0 con 0x7f923c06bd50 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.954+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923c1aceb0 con 0x7f923c06bd50 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.954+0000 7f9238c00640 1 -- 172.21.15.39:0/736415344 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c010690 con 0x7f923c06bd50 2024-09-21T01:01:09.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.954+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f923c1ad360 con 0x7f923c06bd50 2024-09-21T01:01:09.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.957+0000 7f9248c44740 1 Processor -- start 2024-09-21T01:01:09.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.957+0000 7f9248c44740 1 -- start start 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9248c44740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9248c44740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7a89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9238c00640 1 -- 172.21.15.39:0/736415344 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f922c004000 con 0x7f923c06bd50 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9248c44740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7a8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9248c44740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56090b71fd50 con 0x56090b401f60 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9248c44740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56090b70f180 con 0x56090b3e1280 2024-09-21T01:01:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9248c44740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56090b712f90 con 0x56090b3d90a0 2024-09-21T01:01:09.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f923be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f923be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7a8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52860/0 (socket says 172.21.15.39:52860) 2024-09-21T01:01:09.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f923be00640 1 -- 172.21.15.39:0/3249641433 learned_addr learned my addr 172.21.15.39:0/3249641433 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:09.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f9233e00640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7a8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.958+0000 7f923b400640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7a89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923be00640 1 -- 172.21.15.39:0/3249641433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 msgr2=0x56090b7a89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923be00640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7a89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923be00640 1 -- 172.21.15.39:0/3249641433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 msgr2=0x56090b7a8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923be00640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7a8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923be00640 1 -- 172.21.15.39:0/3249641433 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56090b6d9a70 con 0x56090b401f60 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f9233e00640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7a8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:09.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923b400640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7a89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f923be00640 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7a8490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9234009be0 tx=0x7f9234030430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e9bf0c0656c4d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f922be00640 1 -- 172.21.15.39:0/3249641433 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234041030 con 0x56090b401f60 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f922be00640 1 -- 172.21.15.39:0/3249641433 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9234003e20 con 0x56090b401f60 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.959+0000 7f922be00640 1 -- 172.21.15.39:0/3249641433 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234036040 con 0x56090b401f60 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.960+0000 7f9238c00640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f92200b6560 0x7f92200b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.960+0000 7f9238c00640 1 -- 172.21.15.39:0/736415344 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f922c0e7b10 con 0x7f923c06bd50 2024-09-21T01:01:09.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.960+0000 7f923b400640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f92200b6560 0x7f92200b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.960+0000 7f9248c44740 1 -- 172.21.15.39:0/3249641433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 msgr2=0x56090b7a8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.960+0000 7f9248c44740 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7a8490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9234009be0 tx=0x7f9234030430 comp rx=0 tx=0).stop 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 -- 172.21.15.39:0/3249641433 shutdown_connections 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7a8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7a89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 --2- 172.21.15.39:0/3249641433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7a8490 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 -- 172.21.15.39:0/3249641433 >> 172.21.15.39:0/3249641433 conn(0x56090b3f6350 msgr2=0x56090b7a6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f923b400640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f92200b6560 0x7f92200b8a70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f922c0ad510 tx=0x7f922c000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 -- 172.21.15.39:0/3249641433 shutdown_connections 2024-09-21T01:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 -- 172.21.15.39:0/3249641433 wait complete. 2024-09-21T01:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 -- start start 2024-09-21T01:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.961+0000 7f9248c44740 1 Processor -- start 2024-09-21T01:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7bff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56090b5705f0 con 0x56090b401f60 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56090b6aa460 con 0x56090b3e1280 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56090b7b6c10 con 0x56090b3d90a0 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9248c44740 10 client.0 _ll_register_callbacks cb 0x56090b6ba210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f923be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7bff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52876/0 (socket says 172.21.15.39:52876) 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 -- 172.21.15.39:0/1575133120 learned_addr learned my addr 172.21.15.39:0/1575133120 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.962+0000 7f9233e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 -- 172.21.15.39:0/1575133120 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 msgr2=0x56090b7b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 -- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 msgr2=0x56090b7b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 -- 172.21.15.39:0/1575133120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56090b46d540 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f923be00640 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7bff00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9234034040 tx=0x7f923402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55211b890caa6e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f9228c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.963+0000 7f9228c00640 1 -- 172.21.15.39:0/1575133120 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923404a070 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9228c00640 1 -- 172.21.15.39:0/1575133120 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f92340306b0 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56090b6aca70 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9228c00640 1 -- 172.21.15.39:0/1575133120 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9234059050 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9248c44740 10 client.14844 fetch_fsmap learned FSMap version 17 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56090b6c76f0 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9228c00640 1 -- 172.21.15.39:0/1575133120 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56090b6aca70 con 0x56090b401f60 2024-09-21T01:01:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.964+0000 7f9228c00640 1 -- 172.21.15.39:0/1575133120 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f9234004590 con 0x56090b401f60 2024-09-21T01:01:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9248c44740 10 client.14844 fetch_fsmap finished waiting for FSMap version 17 2024-09-21T01:01:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9248c44740 10 client.14844 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:01:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9248c44740 20 client.14844 populate_metadata read hostname 'smithi039' 2024-09-21T01:01:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9248c44740 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b3c08e0 0x56090b7c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9248c44740 4 client.14844 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:01:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56090b721ca0 con 0x56090b3c08e0 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.965+0000 7f9233e00640 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b3c08e0 0x56090b7c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.966+0000 7f9233e00640 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b3c08e0 0x56090b7c6060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=814468235f13184 server_cookie=b3c49d93937accb9 in_seq=0 out_seq=0 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.966+0000 7f9228c00640 10 client.14844 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9228c00640 1 -- 172.21.15.39:0/1575133120 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56090b721ca0 con 0x56090b3c08e0 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9228c00640 10 client.14844 handle_command_reply: tid=0 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 client.14844 shutdown 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 msgr2=0x56090b7bff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7bff00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9234034040 tx=0x7f923402a6c0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 shutdown_connections 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b3c08e0 0x56090b7c6060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b401f60 0x56090b7bff00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 --2- 172.21.15.39:0/1575133120 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9228c00640 0 client.14844 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 >> 172.21.15.39:0/1575133120 conn(0x56090b3f6350 msgr2=0x56090b7ab670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 shutdown_connections 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.967+0000 7f9248c44740 1 -- 172.21.15.39:0/1575133120 wait complete. 2024-09-21T01:01:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.968+0000 7f9248c44740 20 client.14844 trim_cache size 0 max 16384 2024-09-21T01:01:09.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 Processor -- start 2024-09-21T01:01:09.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 -- start start 2024-09-21T01:01:09.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 0x56090b7ae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b401f60 0x56090b7ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56090b7c43a0 con 0x56090b3e1280 2024-09-21T01:01:09.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56090b7c4a70 con 0x56090b401f60 2024-09-21T01:01:09.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9248c44740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56090b7c5140 con 0x56090b3d90a0 2024-09-21T01:01:09.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9233e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 0x56090b7ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9233e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 0x56090b7ae360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52890/0 (socket says 172.21.15.39:52890) 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.987+0000 7f9233e00640 1 -- 172.21.15.39:0/2792822076 learned_addr learned my addr 172.21.15.39:0/2792822076 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f923be00640 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f923be00640 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b401f60 0x56090b7ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f9233e00640 1 -- 172.21.15.39:0/2792822076 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 msgr2=0x56090b7bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f9233e00640 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7bfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f9233e00640 1 -- 172.21.15.39:0/2792822076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b401f60 msgr2=0x56090b7ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f9233e00640 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b401f60 0x56090b7ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f9233e00640 1 -- 172.21.15.39:0/2792822076 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56090b7af730 con 0x56090b3e1280 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f9233e00640 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 0x56090b7ae360 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f923c06b4f0 tx=0x7f9224005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b61522757307e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.988+0000 7f922b400640 1 -- 172.21.15.39:0/2792822076 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224005020 con 0x56090b3e1280 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f922b400640 1 -- 172.21.15.39:0/2792822076 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92240047a0 con 0x56090b3e1280 2024-09-21T01:01:09.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f922b400640 1 -- 172.21.15.39:0/2792822076 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92240107b0 con 0x56090b3e1280 2024-09-21T01:01:09.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 -- 172.21.15.39:0/2792822076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 msgr2=0x56090b7ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 0x56090b7ae360 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f923c06b4f0 tx=0x7f9224005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:09.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 -- 172.21.15.39:0/2792822076 shutdown_connections 2024-09-21T01:01:09.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b401f60 0x56090b7ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3e1280 0x56090b7ae360 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 --2- 172.21.15.39:0/2792822076 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b3d90a0 0x56090b7bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 -- 172.21.15.39:0/2792822076 >> 172.21.15.39:0/2792822076 conn(0x56090b3f6350 msgr2=0x56090b7a1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.989+0000 7f9248c44740 1 -- 172.21.15.39:0/2792822076 shutdown_connections 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.990+0000 7f9248c44740 1 -- 172.21.15.39:0/2792822076 wait complete. 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.990+0000 7f9248c44740 1 -- start start 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.990+0000 7f9248c44740 1 Processor -- start 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3d90a0 0x56090b7cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7c8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b401f60 0x56090b7c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56090b7c5ee0 con 0x56090b3d90a0 2024-09-21T01:01:09.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56090b7c5810 con 0x56090b3e1280 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56090b7c65b0 con 0x56090b401f60 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9248c44740 10 client.0 _ll_register_callbacks cb 0x7f9234004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f923be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3d90a0 0x56090b7cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39546/0 (socket says 172.21.15.39:39546) 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 -- 172.21.15.39:0/3225773221 learned_addr learned my addr 172.21.15.39:0/3225773221 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f923be00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b401f60 0x56090b7c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 -- 172.21.15.39:0/3225773221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b401f60 msgr2=0x56090b7c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b401f60 0x56090b7c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 -- 172.21.15.39:0/3225773221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3d90a0 msgr2=0x56090b7cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3d90a0 0x56090b7cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.991+0000 7f9233e00640 1 -- 172.21.15.39:0/3225773221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56090b7cd8a0 con 0x56090b3e1280 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.992+0000 7f923be00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b401f60 0x56090b7c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:01:09.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.992+0000 7f9233e00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7c8110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9224000c00 tx=0x7f9224009c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfaa4b1d0baca0d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:09.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.992+0000 7f9228c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:01:09.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.992+0000 7f9228c00640 1 -- 172.21.15.39:0/3225773221 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224028040 con 0x56090b3e1280 2024-09-21T01:01:09.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.992+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92240047a0 con 0x56090b3e1280 2024-09-21T01:01:09.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.992+0000 7f9228c00640 1 -- 172.21.15.39:0/3225773221 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f9224004230 con 0x56090b3e1280 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.993+0000 7f9228c00640 1 -- 172.21.15.39:0/3225773221 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9224004470 con 0x56090b3e1280 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.993+0000 7f9248c44740 10 client.24866 fetch_fsmap learned FSMap version 17 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.993+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56090b831670 con 0x56090b3e1280 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.993+0000 7f9228c00640 1 -- 172.21.15.39:0/3225773221 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92240047a0 con 0x56090b3e1280 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.993+0000 7f9228c00640 1 -- 172.21.15.39:0/3225773221 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f9224034030 con 0x56090b3e1280 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9248c44740 10 client.24866 fetch_fsmap finished waiting for FSMap version 17 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9248c44740 10 client.24866 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9248c44740 20 client.24866 populate_metadata read hostname 'smithi039' 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9248c44740 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b395c90 0x56090b834c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9248c44740 4 client.24866 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "status"}) -- 0x56090b71d750 con 0x56090b395c90 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f923be00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b395c90 0x56090b834c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:09.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f923be00640 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b395c90 0x56090b834c70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ffe6b8af094a03e server_cookie=7aad4b74a3b674e7 in_seq=0 out_seq=0 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.994+0000 7f9228c00640 10 client.24866 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9228c00640 1 -- 172.21.15.39:0/3225773221 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56090b71d750 con 0x56090b395c90 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9228c00640 10 client.24866 handle_command_reply: tid=0 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 client.24866 shutdown 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 msgr2=0x56090b7c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7c8110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9224000c00 tx=0x7f9224009c10 comp rx=0 tx=0).stop 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 shutdown_connections 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56090b395c90 0x56090b834c70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56090b401f60 0x56090b7c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56090b3e1280 0x56090b7c8110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 --2- 172.21.15.39:0/3225773221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56090b3d90a0 0x56090b7cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 >> 172.21.15.39:0/3225773221 conn(0x56090b3f6350 msgr2=0x56090b7b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9228c00640 0 client.24866 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 shutdown_connections 2024-09-21T01:01:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 1 -- 172.21.15.39:0/3225773221 wait complete. 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.995+0000 7f9248c44740 20 client.24866 trim_cache size 0 max 16384 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24764, 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 25.495701945, 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 17, 2024-09-21T01:01:10.000 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 88, 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 85, 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 40.509961809000004 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f92200b6560 msgr2=0x7f92200b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f92200b6560 0x7f92200b8a70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f922c0ad510 tx=0x7f922c000bc0 comp rx=0 tx=0).stop 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 msgr2=0x7f923c1043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 0x7f923c1043d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f922c00e9c0 tx=0x7f922c00ee80 comp rx=0 tx=0).stop 2024-09-21T01:01:10.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 shutdown_connections 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f923c106c20 0x7f923c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f923c1062d0 0x7f923c1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f92200b6560 0x7f92200b8a70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 --2- 172.21.15.39:0/736415344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c06bd50 0x7f923c1043d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 >> 172.21.15.39:0/736415344 conn(0x7f923c0799a0 msgr2=0x7f923c107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 shutdown_connections 2024-09-21T01:01:10.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:09.996+0000 7f9242000640 1 -- 172.21.15.39:0/736415344 wait complete. 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-09-21T01:01:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-09-21T01:01:10.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-21T01:01:10.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-09-21T01:01:10.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-21T01:01:10.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:09 smithi150 ceph-mon[34945]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-09-21T01:01:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-21T01:01:10.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-09-21T01:01:10.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-21T01:01:10.108 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:09 smithi160 ceph-mon[34696]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T01:01:10.868 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:10 smithi039 ceph-mon[30415]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T01:01:10.868 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:10 smithi039 ceph-mon[30415]: pgmap v317: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:10.908 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T01:01:10.908 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:10.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T01:01:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:10 smithi150 ceph-mon[34945]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T01:01:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:10 smithi150 ceph-mon[34945]: pgmap v317: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:11.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:10 smithi160 ceph-mon[34696]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T01:01:11.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:10 smithi160 ceph-mon[34696]: pgmap v317: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:11.285 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 -- 172.21.15.39:0/1005765831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc109e10 msgr2=0x7f3cfc110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 --2- 172.21.15.39:0/1005765831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc109e10 0x7f3cfc110640 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009ca0 tx=0x7f3cf402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 -- 172.21.15.39:0/1005765831 shutdown_connections 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 --2- 172.21.15.39:0/1005765831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc109e10 0x7f3cfc110640 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 --2- 172.21.15.39:0/1005765831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1094c0 0x7f3cfc1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 --2- 172.21.15.39:0/1005765831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc103b80 0x7f3cfc103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 -- 172.21.15.39:0/1005765831 >> 172.21.15.39:0/1005765831 conn(0x7f3cfc0ff5d0 msgr2=0x7f3cfc101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 -- 172.21.15.39:0/1005765831 shutdown_connections 2024-09-21T01:01:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.891+0000 7f3d03600640 1 -- 172.21.15.39:0/1005765831 wait complete. 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.892+0000 7f3d03600640 1 Processor -- start 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.892+0000 7f3d03600640 1 -- start start 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d03600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc103b80 0x7f3cfc19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d03600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc1094c0 0x7f3cfc19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d03600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc109e10 0x7f3cfc1a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d03600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc113b20 con 0x7f3cfc103b80 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d03600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc1139a0 con 0x7f3cfc109e10 2024-09-21T01:01:11.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d03600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc113ca0 con 0x7f3cfc1094c0 2024-09-21T01:01:11.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d00c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc103b80 0x7f3cfc19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:11.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d01600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc109e10 0x7f3cfc1a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3cf3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc1094c0 0x7f3cfc19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d01600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc109e10 0x7f3cfc1a1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39564/0 (socket says 172.21.15.39:39564) 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.893+0000 7f3d01600640 1 -- 172.21.15.39:0/444606066 learned_addr learned my addr 172.21.15.39:0/444606066 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cf3e00640 1 -- 172.21.15.39:0/444606066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc109e10 msgr2=0x7f3cfc1a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cf3e00640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc109e10 0x7f3cfc1a1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cf3e00640 1 -- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc103b80 msgr2=0x7f3cfc19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cf3e00640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc103b80 0x7f3cfc19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cf3e00640 1 -- 172.21.15.39:0/444606066 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cfc1a25c0 con 0x7f3cfc1094c0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cf3e00640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc1094c0 0x7f3cfc19cbf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc06ca10 tx=0x7f3ce400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce400cd00 con 0x7f3cfc1094c0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cfc1a28b0 con 0x7f3cfc1094c0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.894+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cfc1a9010 con 0x7f3cfc1094c0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.895+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ce40035f0 con 0x7f3cfc1094c0 2024-09-21T01:01:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.895+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4018690 con 0x7f3cfc1094c0 2024-09-21T01:01:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.896+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd80052c0 con 0x7f3cfc1094c0 2024-09-21T01:01:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.898+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f3ce4004020 con 0x7f3cfc1094c0 2024-09-21T01:01:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.900+0000 7f3cfa000640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ce00b6560 0x7f3ce00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.900+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f3ce40e78e0 con 0x7f3cfc1094c0 2024-09-21T01:01:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.900+0000 7f3d00c00640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ce00b6560 0x7f3ce00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.901+0000 7f3d00c00640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ce00b6560 0x7f3ce00b8a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009bd0 tx=0x7f3cec005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:11.901+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3ce40add70 con 0x7f3cfc1094c0 2024-09-21T01:01:12.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.143+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3cd8005880 con 0x7f3cfc1094c0 2024-09-21T01:01:12.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.144+0000 7f3cfa000640 1 -- 172.21.15.39:0/444606066 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f3ce40b2150 con 0x7f3cfc1094c0 2024-09-21T01:01:12.145 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-21T01:01:12.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ce00b6560 msgr2=0x7f3ce00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:12.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ce00b6560 0x7f3ce00b8a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009bd0 tx=0x7f3cec005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc1094c0 msgr2=0x7f3cfc19cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc1094c0 0x7f3cfc19cbf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc06ca10 tx=0x7f3ce400ee50 comp rx=0 tx=0).stop 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 shutdown_connections 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ce00b6560 0x7f3ce00b8a70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc109e10 0x7f3cfc1a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3cfc1094c0 0x7f3cfc19cbf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 --2- 172.21.15.39:0/444606066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cfc103b80 0x7f3cfc19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 >> 172.21.15.39:0/444606066 conn(0x7f3cfc0ff5d0 msgr2=0x7f3cfc106a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 shutdown_connections 2024-09-21T01:01:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:12.147+0000 7f3d03600640 1 -- 172.21.15.39:0/444606066 wait complete. 2024-09-21T01:01:13.069 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-21T01:01:13.069 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T01:01:13.069 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-09-21T01:01:13.070 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-21T01:01:13.070 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:13.070 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-21T01:01:13.070 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T01:01:13.070 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T01:01:13.070 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-21T01:01:13.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:13.070 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-09-21T01:01:13.105 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T01:01:13.105 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T01:01:13.105 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T01:01:13.105 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T01:01:13.106 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42081sec preferred_lft 42081sec 2024-09-21T01:01:13.107 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link 2024-09-21T01:01:13.107 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:01:13.107 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:01:13.107 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:01:13.107 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-09-21T01:01:13.108 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T01:01:13.108 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:01:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:01:13.243 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:12 smithi039 ceph-mon[30415]: pgmap v318: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:13.243 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:12 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/444606066' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:01:13.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:12 smithi150 ceph-mon[34945]: pgmap v318: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:13.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:12 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/444606066' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:01:13.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:12 smithi160 ceph-mon[34696]: pgmap v318: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:13.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:12 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/444606066' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:01:13.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:13.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:13.550 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T01:01:13.606 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-09-21T01:01:13.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:13.609 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-09-21T01:01:13.653 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-09-21T01:01:13.654 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T01:01:13.654 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T01:01:13.654 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T01:01:13.654 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T01:01:13.654 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:01:13.654 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:01:13.654 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T01:01:13.655 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T01:01:13.655 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T01:01:13.655 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:01:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:01:14.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:14.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:14.338 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-09-21T01:01:14.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:14.375 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-09-21T01:01:14.450 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:01:14.451 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:01:14.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:14.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-21T01:01:14.451 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:01:14.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:01:14.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:14.633 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-21T01:01:14.633 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:01:14.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:01:14.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:14.893 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:01:14.893 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:01:14.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-09-21T01:01:14.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T01:01:14.894 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:01:14.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:01:15.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:15.062 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-21T01:01:15.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:01:15.063 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:15.097 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-21T01:01:15.098 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:01:15.098 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:15.174 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-09-21T01:01:15.274 DEBUG:teuthology.orchestra.run.smithi039:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:01:15.311 INFO:teuthology.orchestra.run.smithi039.stdout:/proc 2024-09-21T01:01:15.311 INFO:teuthology.orchestra.run.smithi039.stdout:/sys 2024-09-21T01:01:15.311 INFO:teuthology.orchestra.run.smithi039.stdout:/dev 2024-09-21T01:01:15.311 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/security 2024-09-21T01:01:15.311 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/shm 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/pts 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/run 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/cgroup 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/pstore 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/bpf 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/ 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/selinux 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/hugepages 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/mqueue 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/debug 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/tracing 2024-09-21T01:01:15.312 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/fuse/connections 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/config 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/1000 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/ceph 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/ff3d53737ee3cc1bc579a318ef1567b986a246647b26489bc1272e42c2b2a54e/merged 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/da16028f57797ec1f8eb6af828a18c6edb0010cadc42039112abe7d2ea41e406/merged 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/0 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2a7c48572411d4d169f3ccd6248889bf14559bbe1bcf94ff0a67a7e4780cbc96/merged 2024-09-21T01:01:15.313 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/3642a673b3016cc1bccb51d914731c5c46c2350147d7368138560951c31066bc/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/0f2da99f6dbb33bdbfe740aedd9f0ce1f4f62091e10d58b343bd7d93050d0e98/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/d85c67adde889546e8b622e007c2d24ee762fd9552e7f5928611ebe9974d29e9/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/0fabe2d51060b59809f1898ff99a8d9bfbfb7db7d012fbd5e03d8c3dcdc95cf4/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/ae98b6e7a62c5743dd5ef5586b7c36d45391eb1c8e944630e77b70eda18136a2/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/1c5cba3a251f0316d2932cb4cbcc6c560d70201a86bed7489f5e6bee9562ae5b/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/dec923e06a6b415dd36cd3471888872ae3c91b4f48b2b44c33b53743986e423c/merged 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:15.314 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:15.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:15.315 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-09-21T01:01:15.391 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T01:01:15.391 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-21T01:01:15.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-09-21T01:01:15.483 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:15 smithi039 ceph-mon[30415]: pgmap v319: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:15.494 DEBUG:teuthology.orchestra.run.smithi039:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:01:15.532 INFO:teuthology.orchestra.run.smithi039.stdout:/proc 2024-09-21T01:01:15.532 INFO:teuthology.orchestra.run.smithi039.stdout:/sys 2024-09-21T01:01:15.532 INFO:teuthology.orchestra.run.smithi039.stdout:/dev 2024-09-21T01:01:15.532 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/security 2024-09-21T01:01:15.532 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/shm 2024-09-21T01:01:15.532 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/pts 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/run 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/cgroup 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/pstore 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/bpf 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/ 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/selinux 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/hugepages 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/mqueue 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/debug 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/tracing 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/fuse/connections 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/config 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:01:15.533 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/1000 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/ceph 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/ff3d53737ee3cc1bc579a318ef1567b986a246647b26489bc1272e42c2b2a54e/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/da16028f57797ec1f8eb6af828a18c6edb0010cadc42039112abe7d2ea41e406/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/0 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2a7c48572411d4d169f3ccd6248889bf14559bbe1bcf94ff0a67a7e4780cbc96/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/3642a673b3016cc1bccb51d914731c5c46c2350147d7368138560951c31066bc/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/0f2da99f6dbb33bdbfe740aedd9f0ce1f4f62091e10d58b343bd7d93050d0e98/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/d85c67adde889546e8b622e007c2d24ee762fd9552e7f5928611ebe9974d29e9/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/0fabe2d51060b59809f1898ff99a8d9bfbfb7db7d012fbd5e03d8c3dcdc95cf4/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/ae98b6e7a62c5743dd5ef5586b7c36d45391eb1c8e944630e77b70eda18136a2/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/1c5cba3a251f0316d2932cb4cbcc6c560d70201a86bed7489f5e6bee9562ae5b/merged 2024-09-21T01:01:15.534 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/dec923e06a6b415dd36cd3471888872ae3c91b4f48b2b44c33b53743986e423c/merged 2024-09-21T01:01:15.535 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns 2024-09-21T01:01:15.535 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:15.535 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:15.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:15.535 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-09-21T01:01:15.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:15 smithi150 ceph-mon[34945]: pgmap v319: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:15.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:15 smithi160 ceph-mon[34696]: pgmap v319: 129 pgs: 4 peering, 125 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:15.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:2024-09-21T01:01:15.694+0000 7f721eb21480 -1 init, newargv = 0x7f72080046a0 newargc=15 2024-09-21T01:01:15.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:ceph-fuse[75360]: starting ceph client 2024-09-21T01:01:15.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:ceph-fuse[75360]: starting fuse 2024-09-21T01:01:16.711 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-09-21T01:01:16.786 DEBUG:teuthology.orchestra.run.smithi039:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/proc 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/sys 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/dev 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/security 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/shm 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/pts 2024-09-21T01:01:16.822 INFO:teuthology.orchestra.run.smithi039.stdout:/run 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/cgroup 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/pstore 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/bpf 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/ 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/selinux 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/hugepages 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/mqueue 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/debug 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/tracing 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/fuse/connections 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/config 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:01:16.823 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/1000 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/ceph 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/ff3d53737ee3cc1bc579a318ef1567b986a246647b26489bc1272e42c2b2a54e/merged 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/da16028f57797ec1f8eb6af828a18c6edb0010cadc42039112abe7d2ea41e406/merged 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/0 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2a7c48572411d4d169f3ccd6248889bf14559bbe1bcf94ff0a67a7e4780cbc96/merged 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/3642a673b3016cc1bccb51d914731c5c46c2350147d7368138560951c31066bc/merged 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/0f2da99f6dbb33bdbfe740aedd9f0ce1f4f62091e10d58b343bd7d93050d0e98/merged 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/d85c67adde889546e8b622e007c2d24ee762fd9552e7f5928611ebe9974d29e9/merged 2024-09-21T01:01:16.824 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/0fabe2d51060b59809f1898ff99a8d9bfbfb7db7d012fbd5e03d8c3dcdc95cf4/merged 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/ae98b6e7a62c5743dd5ef5586b7c36d45391eb1c8e944630e77b70eda18136a2/merged 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/1c5cba3a251f0316d2932cb4cbcc6c560d70201a86bed7489f5e6bee9562ae5b/merged 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/dec923e06a6b415dd36cd3471888872ae3c91b4f48b2b44c33b53743986e423c/merged 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:01:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:/home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:16.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:16.826 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-09-21T01:01:16.901 INFO:teuthology.orchestra.run.smithi039.stdout:120 2024-09-21T01:01:16.901 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo stdin-killer -- python3 -c ' 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-09-21T01:01:16.902 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.admin")) 2024-09-21T01:01:16.903 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-09-21T01:01:17.066 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.admin.75360.asok 2024-09-21T01:01:17.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:17.081 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75360.asok 2024-09-21T01:01:17.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:17.081 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75360.asok status 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "admin", 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "75360", 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-09-21T01:01:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 14862, 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 14862 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3103882042 2024-09-21T01:01:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3103882042 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.14862 192.168.0.1:0/3103882042", 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/3103882042", 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 17, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 88, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-09-21T01:01:17.334 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:01:17.348 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:17 smithi039 ceph-mon[30415]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:17.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:17.350 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-09-21T01:01:17.350 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:17.386 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-09-21T01:01:17.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:17.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:17.386 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:17.483 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-21T01:01:17.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:17 smithi150 ceph-mon[34945]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:17.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T01:01:17.604 INFO:teuthology.orchestra.run.smithi039.stdout:# file: . 2024-09-21T01:01:17.604 INFO:teuthology.orchestra.run.smithi039.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-21T01:01:17.604 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:17.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:01:17.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:17 smithi160 ceph-mon[34696]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:17.612 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:01:17.612 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:01:17.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:01:18.065 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 -- 172.21.15.39:0/2002620835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c01039b0 msgr2=0x7fd4c0103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 --2- 172.21.15.39:0/2002620835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c01039b0 0x7fd4c0103dc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009900 tx=0x7fd4b002a410 comp rx=0 tx=0).stop 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 -- 172.21.15.39:0/2002620835 shutdown_connections 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 --2- 172.21.15.39:0/2002620835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c0109c40 0x7fd4c0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 --2- 172.21.15.39:0/2002620835 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c0109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 --2- 172.21.15.39:0/2002620835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c01039b0 0x7fd4c0103dc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 -- 172.21.15.39:0/2002620835 >> 172.21.15.39:0/2002620835 conn(0x7fd4c00ff400 msgr2=0x7fd4c0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:18.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 -- 172.21.15.39:0/2002620835 shutdown_connections 2024-09-21T01:01:18.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.694+0000 7fd4bf200640 1 -- 172.21.15.39:0/2002620835 wait complete. 2024-09-21T01:01:18.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.696+0000 7fd4bf200640 1 Processor -- start 2024-09-21T01:01:18.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.696+0000 7fd4bf200640 1 -- start start 2024-09-21T01:01:18.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bf200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c01039b0 0x7fd4c019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:18.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bf200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:18.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bf200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c0109c40 0x7fd4c019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:18.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bf200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0113900 con 0x7fd4c0109c40 2024-09-21T01:01:18.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bf200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0113780 con 0x7fd4c01039b0 2024-09-21T01:01:18.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bf200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0113a80 con 0x7fd4c01092f0 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c019a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36468/0 (socket says 172.21.15.39:36468) 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 -- 172.21.15.39:0/1569873995 learned_addr learned my addr 172.21.15.39:0/1569873995 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4be800640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c0109c40 0x7fd4c019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4bde00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c01039b0 0x7fd4c019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 -- 172.21.15.39:0/1569873995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c01039b0 msgr2=0x7fd4c019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:18.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c01039b0 0x7fd4c019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 -- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c0109c40 msgr2=0x7fd4c019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c0109c40 0x7fd4c019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.697+0000 7fd4b7e00640 1 -- 172.21.15.39:0/1569873995 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c01a0220 con 0x7fd4c01092f0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.698+0000 7fd4b7e00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c019a850 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800e9c0 tx=0x7fd4a800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.698+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a800ccc0 con 0x7fd4c01092f0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.698+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c01a0510 con 0x7fd4c01092f0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.698+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4a80054d0 con 0x7fd4c01092f0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.698+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8010430 con 0x7fd4c01092f0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.698+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01a6c70 con 0x7fd4c01092f0 2024-09-21T01:01:18.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.699+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c0103dc0 con 0x7fd4c01092f0 2024-09-21T01:01:18.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.701+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fd4a8005020 con 0x7fd4c01092f0 2024-09-21T01:01:18.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.703+0000 7fd4b6a00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4a40b6560 0x7fd4a40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:18.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.703+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fd4a80e7e90 con 0x7fd4c01092f0 2024-09-21T01:01:18.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.703+0000 7fd4bde00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4a40b6560 0x7fd4a40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:18.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.704+0000 7fd4bde00640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4a40b6560 0x7fd4a40b8a70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0031440 tx=0x7fd4b0009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:18.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.705+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd4a80ae290 con 0x7fd4c01092f0 2024-09-21T01:01:18.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.918+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd4c019b3e0 con 0x7fd4c01092f0 2024-09-21T01:01:18.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.918+0000 7fd4b6a00640 1 -- 172.21.15.39:0/1569873995 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+884 (secure 0 0 0) 0x7fd4c019b3e0 con 0x7fd4c01092f0 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2061408491 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2520758295 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2575866445 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/368668226 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3183273760 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3378583569 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/412418374 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2575866445 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/368668226 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3601289686 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/712381310 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3703447041 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:18.920 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/3953053829 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:18.921 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3894994060 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:18.921 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/3953053829 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:18.921 INFO:teuthology.orchestra.run.smithi039.stderr:listed 15 entries 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4a40b6560 msgr2=0x7fd4a40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4a40b6560 0x7fd4a40b8a70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0031440 tx=0x7fd4b0009560 comp rx=0 tx=0).stop 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 msgr2=0x7fd4c019a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c019a850 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800e9c0 tx=0x7fd4a800ee80 comp rx=0 tx=0).stop 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 shutdown_connections 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4c0109c40 0x7fd4c019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c01092f0 0x7fd4c019a850 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd4a40b6560 0x7fd4a40b8a70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 --2- 172.21.15.39:0/1569873995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c01039b0 0x7fd4c019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:18.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 >> 172.21.15.39:0/1569873995 conn(0x7fd4c00ff400 msgr2=0x7fd4c0101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:18.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 shutdown_connections 2024-09-21T01:01:18.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:18.921+0000 7fd4bf200640 1 -- 172.21.15.39:0/1569873995 wait complete. 2024-09-21T01:01:19.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:19 smithi150 ceph-mon[34945]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:19.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:19 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1569873995' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:01:19.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:19 smithi160 ceph-mon[34696]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:19.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:19 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1569873995' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:01:19.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:19 smithi039 ceph-mon[30415]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:19.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:19 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1569873995' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:01:19.777 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:01:19.777 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:01:19.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:01:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:20.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 -- 172.21.15.39:0/3541378615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 msgr2=0x7f7c941096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:20.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 --2- 172.21.15.39:0/3541378615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c941096e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009900 tx=0x7f7c7c02a410 comp rx=0 tx=0).stop 2024-09-21T01:01:20.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 -- 172.21.15.39:0/3541378615 shutdown_connections 2024-09-21T01:01:20.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 --2- 172.21.15.39:0/3541378615 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 0x7f7c94110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 --2- 172.21.15.39:0/3541378615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c941096e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 --2- 172.21.15.39:0/3541378615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c94103990 0x7f7c94103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 -- 172.21.15.39:0/3541378615 >> 172.21.15.39:0/3541378615 conn(0x7f7c940ff440 msgr2=0x7f7c94101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 -- 172.21.15.39:0/3541378615 shutdown_connections 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.921+0000 7f7c99c00640 1 -- 172.21.15.39:0/3541378615 wait complete. 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.922+0000 7f7c99c00640 1 Processor -- start 2024-09-21T01:01:20.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 -- start start 2024-09-21T01:01:20.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c94103990 0x7f7c94075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:20.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c94075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 0x7f7c94076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94113930 con 0x7f7c941092d0 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c941137b0 con 0x7f7c94103990 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94113ab0 con 0x7f7c94109c20 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c8b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c94075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c99200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 0x7f7c94076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c8b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c94075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55058/0 (socket says 172.21.15.39:55058) 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c8b400640 1 -- 172.21.15.39:0/814442283 learned_addr learned my addr 172.21.15.39:0/814442283 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.923+0000 7f7c93e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c94103990 0x7f7c94075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c99200640 1 -- 172.21.15.39:0/814442283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c94103990 msgr2=0x7f7c94075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c99200640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c94103990 0x7f7c94075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c99200640 1 -- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 msgr2=0x7f7c94075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c99200640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c94075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c99200640 1 -- 172.21.15.39:0/814442283 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c941a85a0 con 0x7f7c94109c20 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c8b400640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c94075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.924+0000 7f7c99200640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 0x7f7c94076190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009fd0 tx=0x7f7c8c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.925+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00ce20 con 0x7f7c94109c20 2024-09-21T01:01:20.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.925+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c005020 con 0x7f7c94109c20 2024-09-21T01:01:20.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.925+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c941a8890 con 0x7f7c94109c20 2024-09-21T01:01:20.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.925+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c010690 con 0x7f7c94109c20 2024-09-21T01:01:20.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.925+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c941a8df0 con 0x7f7c94109c20 2024-09-21T01:01:20.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.926+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c94103da0 con 0x7f7c94109c20 2024-09-21T01:01:20.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.928+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f7c8c0054d0 con 0x7f7c94109c20 2024-09-21T01:01:20.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.930+0000 7f7c91600640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c780b65e0 0x7f7c780b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:20.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.930+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f7c8c0e7e30 con 0x7f7c94109c20 2024-09-21T01:01:20.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.930+0000 7f7c93e00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c780b65e0 0x7f7c780b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:20.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.931+0000 7f7c93e00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c780b65e0 0x7f7c780b8af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c94077160 tx=0x7f7c84005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:20.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:20.932+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7c8c0ae230 con 0x7f7c94109c20 2024-09-21T01:01:21.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.144+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7c94076a10 con 0x7f7c94109c20 2024-09-21T01:01:21.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.145+0000 7f7c91600640 1 -- 172.21.15.39:0/814442283 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+884 (secure 0 0 0) 0x7f7c8c0b2610 con 0x7f7c94109c20 2024-09-21T01:01:21.146 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2061408491 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:21.146 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2520758295 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:21.146 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2575866445 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/368668226 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3183273760 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3378583569 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/412418374 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2575866445 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/368668226 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3601289686 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/712381310 2024-09-22T00:51:58.924468+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3703447041 2024-09-22T00:51:28.880618+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/3953053829 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3894994060 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/3953053829 2024-09-22T00:53:07.774291+0000 2024-09-21T01:01:21.147 INFO:teuthology.orchestra.run.smithi039.stderr:listed 15 entries 2024-09-21T01:01:21.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c780b65e0 msgr2=0x7f7c780b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:21.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c780b65e0 0x7f7c780b8af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c94077160 tx=0x7f7c84005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:21.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 msgr2=0x7f7c94076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:21.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 0x7f7c94076190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009fd0 tx=0x7f7c8c00ef10 comp rx=0 tx=0).stop 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 shutdown_connections 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c94109c20 0x7f7c94076190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c941092d0 0x7f7c94075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c780b65e0 0x7f7c780b8af0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 --2- 172.21.15.39:0/814442283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c94103990 0x7f7c94075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 >> 172.21.15.39:0/814442283 conn(0x7f7c940ff440 msgr2=0x7f7c94108950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 shutdown_connections 2024-09-21T01:01:21.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:21.148+0000 7f7c99c00640 1 -- 172.21.15.39:0/814442283 wait complete. 2024-09-21T01:01:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:21 smithi039 ceph-mon[30415]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 401 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:21.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:21 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/814442283' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:01:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:21 smithi150 ceph-mon[34945]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 401 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:21 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/814442283' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:01:21.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:21 smithi160 ceph-mon[34696]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 401 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-09-21T01:01:21.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:21 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/814442283' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:01:21.962 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-09-21T01:01:21.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:21.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-21T01:01:22.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:ceph-fuse[75360]: fuse finished with error 0 and tester_r 0 2024-09-21T01:01:22.025 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T01:01:23.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:23 smithi150 ceph-mon[34945]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T01:01:23.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:23 smithi160 ceph-mon[34696]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T01:01:23.617 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:23 smithi039 ceph-mon[30415]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T01:01:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:25 smithi150 ceph-mon[34945]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T01:01:25.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:25 smithi160 ceph-mon[34696]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T01:01:25.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:25 smithi039 ceph-mon[30415]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T01:01:27.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:27 smithi150 ceph-mon[34945]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:27.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:27 smithi160 ceph-mon[34696]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:27 smithi039 ceph-mon[30415]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:28.127 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:01:28.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:01:28.127 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-21T01:01:28.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:01:28.563 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:29.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 -- 172.21.15.39:0/2138659393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 msgr2=0x7ffb48106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 --2- 172.21.15.39:0/2138659393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb48106190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009900 tx=0x7ffb3c02a410 comp rx=0 tx=0).stop 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 -- 172.21.15.39:0/2138659393 shutdown_connections 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 --2- 172.21.15.39:0/2138659393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 0x7ffb48112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 --2- 172.21.15.39:0/2138659393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb4810b6c0 0x7ffb4810bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 --2- 172.21.15.39:0/2138659393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb48106190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 -- 172.21.15.39:0/2138659393 >> 172.21.15.39:0/2138659393 conn(0x7ffb48101810 msgr2=0x7ffb48103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 -- 172.21.15.39:0/2138659393 shutdown_connections 2024-09-21T01:01:29.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.211+0000 7ffb50600640 1 -- 172.21.15.39:0/2138659393 wait complete. 2024-09-21T01:01:29.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.212+0000 7ffb50600640 1 Processor -- start 2024-09-21T01:01:29.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.212+0000 7ffb50600640 1 -- start start 2024-09-21T01:01:29.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb50600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb4819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:29.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb50600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb4810b6c0 0x7ffb4819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb50600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 0x7ffb481a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb50600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48115cd0 con 0x7ffb48105d80 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb50600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48115b50 con 0x7ffb4810c010 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb50600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48115e50 con 0x7ffb4810b6c0 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb4dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb4819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb4dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb4819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38720/0 (socket says 172.21.15.39:38720) 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb4dc00640 1 -- 172.21.15.39:0/4243202225 learned_addr learned my addr 172.21.15.39:0/4243202225 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb4d200640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb4810b6c0 0x7ffb4819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:29.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.213+0000 7ffb4e600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 0x7ffb481a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:29.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4dc00640 1 -- 172.21.15.39:0/4243202225 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb4810b6c0 msgr2=0x7ffb4819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:29.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4dc00640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb4810b6c0 0x7ffb4819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4dc00640 1 -- 172.21.15.39:0/4243202225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 msgr2=0x7ffb481a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:29.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4dc00640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 0x7ffb481a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4dc00640 1 -- 172.21.15.39:0/4243202225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb481a26d0 con 0x7ffb48105d80 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4e600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 0x7ffb481a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.214+0000 7ffb4dc00640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb4819c760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c005940 tx=0x7ffb3c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.215+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c03f030 con 0x7ffb48105d80 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.215+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb3c0044c0 con 0x7ffb48105d80 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.215+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb481a2960 con 0x7ffb48105d80 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.215+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c035070 con 0x7ffb48105d80 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.215+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb481a9120 con 0x7ffb48105d80 2024-09-21T01:01:29.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.216+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb48106190 con 0x7ffb48105d80 2024-09-21T01:01:29.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.220+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ffb3c042030 con 0x7ffb48105d80 2024-09-21T01:01:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.223+0000 7ffb46000640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ffb300b65e0 0x7ffb300b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.223+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7ffb3c107d40 con 0x7ffb48105d80 2024-09-21T01:01:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.223+0000 7ffb4d200640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ffb300b65e0 0x7ffb300b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.224+0000 7ffb4d200640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ffb300b65e0 0x7ffb300b8af0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffb4810b220 tx=0x7ffb34005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:29.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.226+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ffb3c0d2520 con 0x7ffb48105d80 2024-09-21T01:01:29.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:29 smithi039 ceph-mon[30415]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.425+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb4810de40 con 0x7ffb48105d80 2024-09-21T01:01:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.427+0000 7ffb46000640 1 -- 172.21.15.39:0/4243202225 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25143 (secure 0 0 0) 0x7ffb3c0f10c0 con 0x7ffb48105d80 2024-09-21T01:01:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:29.430 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:01:09.673169+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:01:29.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.429+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ffb300b65e0 msgr2=0x7ffb300b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:29.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.429+0000 7ffb50600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ffb300b65e0 0x7ffb300b8af0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffb4810b220 tx=0x7ffb34005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:29.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.429+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 msgr2=0x7ffb4819c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:29.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.429+0000 7ffb50600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb4819c760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c005940 tx=0x7ffb3c0040c0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 shutdown_connections 2024-09-21T01:01:29.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ffb300b65e0 0x7ffb300b8af0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb4810c010 0x7ffb481a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb4810b6c0 0x7ffb4819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 --2- 172.21.15.39:0/4243202225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb48105d80 0x7ffb4819c760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:29.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 >> 172.21.15.39:0/4243202225 conn(0x7ffb48101810 msgr2=0x7ffb48103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:29.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 shutdown_connections 2024-09-21T01:01:29.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:29.430+0000 7ffb50600640 1 -- 172.21.15.39:0/4243202225 wait complete. 2024-09-21T01:01:29.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:29 smithi150 ceph-mon[34945]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:29.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:29 smithi160 ceph-mon[34696]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:30.185 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:01:30.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:30 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4243202225' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:30.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:30 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4243202225' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:30.570 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:30.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:30 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4243202225' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:31.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3635524753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec8105bb0 msgr2=0x7f4ec8105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:31.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3635524753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec8105bb0 0x7f4ec8105fc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009900 tx=0x7f4eb802a410 comp rx=0 tx=0).stop 2024-09-21T01:01:31.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3635524753 shutdown_connections 2024-09-21T01:01:31.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3635524753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec810be40 0x7f4ec8112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3635524753 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4ec810b4f0 0x7f4ec810b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3635524753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec8105bb0 0x7f4ec8105fc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.176+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3635524753 >> 172.21.15.39:0/3635524753 conn(0x7f4ec8101680 msgr2=0x7f4ec8103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:31.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.177+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3635524753 shutdown_connections 2024-09-21T01:01:31.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.177+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3635524753 wait complete. 2024-09-21T01:01:31.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 Processor -- start 2024-09-21T01:01:31.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 -- start start 2024-09-21T01:01:31.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4ec8105bb0 0x7f4ec819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:31.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 0x7f4ec819caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:31.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec810be40 0x7f4ec81a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8115b00 con 0x7f4ec810b4f0 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8115980 con 0x7f4ec810be40 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.178+0000 7f4ec7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8115c80 con 0x7f4ec8105bb0 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 0x7f4ec819caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 0x7f4ec819caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38730/0 (socket says 172.21.15.39:38730) 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 -- 172.21.15.39:0/3924253536 learned_addr learned my addr 172.21.15.39:0/3924253536 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6a00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4ec8105bb0 0x7f4ec819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec7400640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec810be40 0x7f4ec81a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 -- 172.21.15.39:0/3924253536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4ec8105bb0 msgr2=0x7f4ec819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4ec8105bb0 0x7f4ec819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 -- 172.21.15.39:0/3924253536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec810be40 msgr2=0x7f4ec81a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec810be40 0x7f4ec81a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.179+0000 7f4ec6000640 1 -- 172.21.15.39:0/3924253536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec81a24c0 con 0x7f4ec810b4f0 2024-09-21T01:01:31.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.180+0000 7f4ec6000640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 0x7f4ec819caf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400e9c0 tx=0x7f4eb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:31.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.180+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb400ccc0 con 0x7f4ec810b4f0 2024-09-21T01:01:31.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.180+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb40054d0 con 0x7f4ec810b4f0 2024-09-21T01:01:31.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.180+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec81a27b0 con 0x7f4ec810b4f0 2024-09-21T01:01:31.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.180+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec81a8f10 con 0x7f4ec810b4f0 2024-09-21T01:01:31.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.180+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4010430 con 0x7f4ec810b4f0 2024-09-21T01:01:31.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.181+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec8105fc0 con 0x7f4ec810b4f0 2024-09-21T01:01:31.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.184+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f4eb4005000 con 0x7f4ec810b4f0 2024-09-21T01:01:31.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.185+0000 7f4ebf400640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4eac0b6560 0x7f4eac0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:31.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.185+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f4eb40e7d00 con 0x7f4ec810b4f0 2024-09-21T01:01:31.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.185+0000 7f4ec6a00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4eac0b6560 0x7f4eac0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:31.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.186+0000 7f4ec6a00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4eac0b6560 0x7f4eac0b8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8031440 tx=0x7f4eb8009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:31.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.187+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4eb4056710 con 0x7f4ec810b4f0 2024-09-21T01:01:31.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.387+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ec819d680 con 0x7f4ec810b4f0 2024-09-21T01:01:31.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.388+0000 7f4ebf400640 1 -- 172.21.15.39:0/3924253536 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25143 (secure 0 0 0) 0x7f4ec819d680 con 0x7f4ec810b4f0 2024-09-21T01:01:31.390 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:31.390 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:01:09.673169+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:01:31.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4eac0b6560 msgr2=0x7f4eac0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:31.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4eac0b6560 0x7f4eac0b8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8031440 tx=0x7f4eb8009560 comp rx=0 tx=0).stop 2024-09-21T01:01:31.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 msgr2=0x7f4ec819caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:31.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 0x7f4ec819caf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400e9c0 tx=0x7f4eb400ee80 comp rx=0 tx=0).stop 2024-09-21T01:01:31.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 shutdown_connections 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4eac0b6560 0x7f4eac0b8a70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec810be40 0x7f4ec81a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ec810b4f0 0x7f4ec819caf0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 --2- 172.21.15.39:0/3924253536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4ec8105bb0 0x7f4ec819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 >> 172.21.15.39:0/3924253536 conn(0x7f4ec8101680 msgr2=0x7f4ec8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 shutdown_connections 2024-09-21T01:01:31.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:31.391+0000 7f4ec7e00640 1 -- 172.21.15.39:0/3924253536 wait complete. 2024-09-21T01:01:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:31 smithi150 ceph-mon[34945]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:31.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:31 smithi160 ceph-mon[34696]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:31 smithi039 ceph-mon[30415]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T01:01:32.218 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs set cephfs standby_count_wanted 0 2024-09-21T01:01:32.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:32 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3924253536' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:32 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3924253536' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:32.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:32 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3924253536' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:01:32.618 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:33.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.293+0000 7ff486c00640 1 -- 172.21.15.39:0/2898290484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4800760a0 msgr2=0x7ff48010fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.293+0000 7ff486c00640 1 --2- 172.21.15.39:0/2898290484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4800760a0 0x7ff48010fa10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009900 tx=0x7ff46c02a410 comp rx=0 tx=0).stop 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 -- 172.21.15.39:0/2898290484 shutdown_connections 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 --2- 172.21.15.39:0/2898290484 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff48010ff50 0x7ff480112370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 --2- 172.21.15.39:0/2898290484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4800760a0 0x7ff48010fa10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 --2- 172.21.15.39:0/2898290484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480075750 0x7ff480075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 -- 172.21.15.39:0/2898290484 >> 172.21.15.39:0/2898290484 conn(0x7ff4800ff3e0 msgr2=0x7ff480101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 -- 172.21.15.39:0/2898290484 shutdown_connections 2024-09-21T01:01:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.294+0000 7ff486c00640 1 -- 172.21.15.39:0/2898290484 wait complete. 2024-09-21T01:01:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.295+0000 7ff486c00640 1 Processor -- start 2024-09-21T01:01:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.295+0000 7ff486c00640 1 -- start start 2024-09-21T01:01:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.295+0000 7ff486c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff480075750 0x7ff48019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.295+0000 7ff486c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 0x7ff48019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.295+0000 7ff486c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff48010ff50 0x7ff4801a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff486c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff480115b10 con 0x7ff48010ff50 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff486c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff480115990 con 0x7ff4800760a0 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff486c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff480115c90 con 0x7ff480075750 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff486200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff48010ff50 0x7ff4801a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 0x7ff48019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff485800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff480075750 0x7ff48019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 0x7ff48019cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46058/0 (socket says 172.21.15.39:46058) 2024-09-21T01:01:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 -- 172.21.15.39:0/109492606 learned_addr learned my addr 172.21.15.39:0/109492606 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 -- 172.21.15.39:0/109492606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff480075750 msgr2=0x7ff48019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff480075750 0x7ff48019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 -- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff48010ff50 msgr2=0x7ff4801a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff48010ff50 0x7ff4801a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff484e00640 1 -- 172.21.15.39:0/109492606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4801a24b0 con 0x7ff4800760a0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.296+0000 7ff486200640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff48010ff50 0x7ff4801a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.297+0000 7ff484e00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 0x7ff48019cae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff46c005ab0 tx=0x7ff46c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.297+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c03f070 con 0x7ff4800760a0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.297+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4801a2740 con 0x7ff4800760a0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.297+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff46c031b50 con 0x7ff4800760a0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.297+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c035030 con 0x7ff4800760a0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.297+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4801a8f00 con 0x7ff4800760a0 2024-09-21T01:01:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.299+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff480075b60 con 0x7ff4800760a0 2024-09-21T01:01:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.301+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff46c042070 con 0x7ff4800760a0 2024-09-21T01:01:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.303+0000 7ff47e000640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4680b65e0 0x7ff4680b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.303+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7ff46c108110 con 0x7ff4800760a0 2024-09-21T01:01:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.303+0000 7ff485800640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4680b65e0 0x7ff4680b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.304+0000 7ff485800640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4680b65e0 0x7ff4680b8af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff474009bd0 tx=0x7ff474005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.305+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff46c0d28f0 con 0x7ff4800760a0 2024-09-21T01:01:33.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:33 smithi039 ceph-mon[30415]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:33.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:33.542+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff48019d670 con 0x7ff4800760a0 2024-09-21T01:01:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:33 smithi150 ceph-mon[34945]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:33.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:33 smithi160 ceph-mon[34696]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:34.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.306+0000 7ff47e000640 1 -- 172.21.15.39:0/109492606 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7ff46c0efe20 con 0x7ff4800760a0 2024-09-21T01:01:34.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4680b65e0 msgr2=0x7ff4680b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:34.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4680b65e0 0x7ff4680b8af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff474009bd0 tx=0x7ff474005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 msgr2=0x7ff48019cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 0x7ff48019cae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff46c005ab0 tx=0x7ff46c003b90 comp rx=0 tx=0).stop 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 shutdown_connections 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff48010ff50 0x7ff4801a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4680b65e0 0x7ff4680b8af0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4800760a0 0x7ff48019cae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 --2- 172.21.15.39:0/109492606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff480075750 0x7ff48019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 >> 172.21.15.39:0/109492606 conn(0x7ff4800ff3e0 msgr2=0x7ff480109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 shutdown_connections 2024-09-21T01:01:34.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:34.309+0000 7ff486c00640 1 -- 172.21.15.39:0/109492606 wait complete. 2024-09-21T01:01:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:34 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/109492606' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T01:01:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:34 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T01:01:34.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:34 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/109492606' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T01:01:34.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:34 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T01:01:34.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:34 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/109492606' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T01:01:34.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:34 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T01:01:35.109 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-09-21T01:01:35.109 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs set cephfs max_mds 3 2024-09-21T01:01:35.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:35 smithi039 ceph-mon[30415]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:35.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:35 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T01:01:35.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:35 smithi039 ceph-mon[30415]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:01:35.495 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:35 smithi150 ceph-mon[34945]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:35 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T01:01:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:35 smithi150 ceph-mon[34945]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:01:35.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:35 smithi160 ceph-mon[34696]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:35 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T01:01:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:35 smithi160 ceph-mon[34696]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 -- 172.21.15.39:0/2583899984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c103990 msgr2=0x7fc72c103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 --2- 172.21.15.39:0/2583899984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c103990 0x7fc72c103da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009900 tx=0x7fc71c02a410 comp rx=0 tx=0).stop 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 -- 172.21.15.39:0/2583899984 shutdown_connections 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 --2- 172.21.15.39:0/2583899984 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c109c20 0x7fc72c110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 --2- 172.21.15.39:0/2583899984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c1092d0 0x7fc72c1096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 --2- 172.21.15.39:0/2583899984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c103990 0x7fc72c103da0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 -- 172.21.15.39:0/2583899984 >> 172.21.15.39:0/2583899984 conn(0x7fc72c0ff440 msgr2=0x7fc72c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:36.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 -- 172.21.15.39:0/2583899984 shutdown_connections 2024-09-21T01:01:36.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.189+0000 7fc731200640 1 -- 172.21.15.39:0/2583899984 wait complete. 2024-09-21T01:01:36.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.191+0000 7fc731200640 1 Processor -- start 2024-09-21T01:01:36.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.191+0000 7fc731200640 1 -- start start 2024-09-21T01:01:36.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.191+0000 7fc731200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c103990 0x7fc72c19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:36.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.191+0000 7fc731200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 0x7fc72c19cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc731200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 0x7fc72c1a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc731200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1138e0 con 0x7fc72c109c20 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc731200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c113760 con 0x7fc72c103990 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc731200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c113a60 con 0x7fc72c1092d0 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc72be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 0x7fc72c1a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 0x7fc72c19cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 0x7fc72c19cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:56902/0 (socket says 172.21.15.39:56902) 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc72be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 0x7fc72c1a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59774/0 (socket says 172.21.15.39:59774) 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 -- 172.21.15.39:0/3523642601 learned_addr learned my addr 172.21.15.39:0/3523642601 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc72b400640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c103990 0x7fc72c19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:36.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 -- 172.21.15.39:0/3523642601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c103990 msgr2=0x7fc72c19c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:36.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c103990 0x7fc72c19c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:36.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 -- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 msgr2=0x7fc72c1a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:36.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 0x7fc72c1a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:36.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc722a00640 1 -- 172.21.15.39:0/3523642601 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72c1a24e0 con 0x7fc72c1092d0 2024-09-21T01:01:36.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.192+0000 7fc72be00640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 0x7fc72c1a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.193+0000 7fc722a00640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 0x7fc72c19cb10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc71400e9f0 tx=0x7fc71400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.193+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71400ccc0 con 0x7fc72c1092d0 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.193+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72c1a27d0 con 0x7fc72c1092d0 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.193+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7140108e0 con 0x7fc72c1092d0 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.193+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714018860 con 0x7fc72c1092d0 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.193+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72c1a8f30 con 0x7fc72c1092d0 2024-09-21T01:01:36.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.194+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc72c103da0 con 0x7fc72c1092d0 2024-09-21T01:01:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.197+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fc714010430 con 0x7fc72c1092d0 2024-09-21T01:01:36.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.198+0000 7fc728c00640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7100b6560 0x7fc7100b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:36.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.199+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fc7140e7f20 con 0x7fc72c1092d0 2024-09-21T01:01:36.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.199+0000 7fc72b400640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7100b6560 0x7fc7100b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:36.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.199+0000 7fc72b400640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7100b6560 0x7fc7100b8a70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc71c031440 tx=0x7fc71c009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:36.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.200+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc7140b2700 con 0x7fc72c1092d0 2024-09-21T01:01:36.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:36.437+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fc72c1a91d0 con 0x7fc72c1092d0 2024-09-21T01:01:37.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.321+0000 7fc728c00640 1 -- 172.21.15.39:0/3523642601 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fc7140b7640 con 0x7fc72c1092d0 2024-09-21T01:01:37.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7100b6560 msgr2=0x7fc7100b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:37.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7100b6560 0x7fc7100b8a70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc71c031440 tx=0x7fc71c009560 comp rx=0 tx=0).stop 2024-09-21T01:01:37.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 msgr2=0x7fc72c19cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 0x7fc72c19cb10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc71400e9f0 tx=0x7fc71400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 shutdown_connections 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc72c109c20 0x7fc72c1a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc72c1092d0 0x7fc72c19cb10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7100b6560 0x7fc7100b8a70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 --2- 172.21.15.39:0/3523642601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c103990 0x7fc72c19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:37.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 >> 172.21.15.39:0/3523642601 conn(0x7fc72c0ff440 msgr2=0x7fc72c108940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:37.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 shutdown_connections 2024-09-21T01:01:37.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:37.325+0000 7fc731200640 1 -- 172.21.15.39:0/3523642601 wait complete. 2024-09-21T01:01:37.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:37 smithi150 ceph-mon[34945]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:37.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:37 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3523642601' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} : dispatch 2024-09-21T01:01:37.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:37 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} : dispatch 2024-09-21T01:01:37.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:37 smithi160 ceph-mon[34696]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:37.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:37 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3523642601' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} : dispatch 2024-09-21T01:01:37.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:37 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} : dispatch 2024-09-21T01:01:37.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:37 smithi039 ceph-mon[30415]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T01:01:37.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:37 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3523642601' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} : dispatch 2024-09-21T01:01:37.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:37 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} : dispatch 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-09-21T01:01:38.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:38 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:01:38.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs set cephfs allow_standby_replay true 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-09-21T01:01:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:38 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:01:38.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:01:38.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-21T01:01:38.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T01:01:38.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T01:01:38.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-09-21T01:01:38.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-09-21T01:01:38.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:38 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:01:39.425 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] up:active 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: Cluster is now healthy 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.634 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:39.634 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] up:active 2024-09-21T01:01:39.634 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: Cluster is now healthy 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.635 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] up:active 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: Cluster is now healthy 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:39.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:40.064 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: 2024-09-21T01:01:39.632+0000 7f8f1b130640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-21T01:01:40.064 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:01:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: 2024-09-21T01:01:39.632+0000 7f8f1b130640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 -- 172.21.15.39:0/732121310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301039b0 msgr2=0x7fe330103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 --2- 172.21.15.39:0/732121310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301039b0 0x7fe330103dc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe324009900 tx=0x7fe32402a410 comp rx=0 tx=0).stop 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 -- 172.21.15.39:0/732121310 shutdown_connections 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 --2- 172.21.15.39:0/732121310 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe330109c40 0x7fe330110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 --2- 172.21.15.39:0/732121310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301092f0 0x7fe330109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 --2- 172.21.15.39:0/732121310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301039b0 0x7fe330103dc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:40.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 -- 172.21.15.39:0/732121310 >> 172.21.15.39:0/732121310 conn(0x7fe3300ff440 msgr2=0x7fe330101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:40.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 -- 172.21.15.39:0/732121310 shutdown_connections 2024-09-21T01:01:40.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.127+0000 7fe336c00640 1 -- 172.21.15.39:0/732121310 wait complete. 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.128+0000 7fe336c00640 1 Processor -- start 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 -- start start 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301039b0 0x7fe33019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301092f0 0x7fe33019a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe330109c40 0x7fe33019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe330113950 con 0x7fe3301092f0 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301137d0 con 0x7fe3301039b0 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe336c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe330113ad0 con 0x7fe330109c40 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe334e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301092f0 0x7fe33019a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe334e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301092f0 0x7fe33019a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59792/0 (socket says 172.21.15.39:59792) 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe334e00640 1 -- 172.21.15.39:0/132250288 learned_addr learned my addr 172.21.15.39:0/132250288 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.129+0000 7fe335800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301039b0 0x7fe33019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:40.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe336200640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe330109c40 0x7fe33019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:40.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe335800640 1 -- 172.21.15.39:0/132250288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe330109c40 msgr2=0x7fe33019fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:40.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe335800640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe330109c40 0x7fe33019fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:40.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe335800640 1 -- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301092f0 msgr2=0x7fe33019a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:40.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe335800640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301092f0 0x7fe33019a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe335800640 1 -- 172.21.15.39:0/132250288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3301a0250 con 0x7fe3301039b0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.130+0000 7fe335800640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301039b0 0x7fe33019a2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe32402aa00 tx=0x7fe324003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.131+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32403f070 con 0x7fe3301039b0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.131+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3301a04e0 con 0x7fe3301039b0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.131+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe324031b50 con 0x7fe3301039b0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.131+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324035030 con 0x7fe3301039b0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.131+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3301a6ca0 con 0x7fe3301039b0 2024-09-21T01:01:40.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.132+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe330104c30 con 0x7fe3301039b0 2024-09-21T01:01:40.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.134+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fe324042070 con 0x7fe3301039b0 2024-09-21T01:01:40.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.136+0000 7fe32e000640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3180b6560 0x7fe3180b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:40.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.136+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fe324107f70 con 0x7fe3301039b0 2024-09-21T01:01:40.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.136+0000 7fe334e00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3180b6560 0x7fe3180b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:40.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.137+0000 7fe334e00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3180b6560 0x7fe3180b8a70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0095f0 tx=0x7fe31c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:40.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.138+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe324031d00 con 0x7fe3301039b0 2024-09-21T01:01:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:40.389+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fe330103dc0 con 0x7fe3301039b0 2024-09-21T01:01:40.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:40 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] up:active 2024-09-21T01:01:40.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:40 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-09-21T01:01:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:40 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] up:active 2024-09-21T01:01:40.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:40 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-09-21T01:01:40.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:40 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] up:active 2024-09-21T01:01:40.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:40 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-09-21T01:01:41.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.363+0000 7fe32e000640 1 -- 172.21.15.39:0/132250288 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7fe3240d2750 con 0x7fe3301039b0 2024-09-21T01:01:41.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3180b6560 msgr2=0x7fe3180b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:41.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3180b6560 0x7fe3180b8a70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0095f0 tx=0x7fe31c005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:41.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301039b0 msgr2=0x7fe33019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:41.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301039b0 0x7fe33019a2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe32402aa00 tx=0x7fe324003b90 comp rx=0 tx=0).stop 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 shutdown_connections 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe330109c40 0x7fe33019fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3301092f0 0x7fe33019a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3180b6560 0x7fe3180b8a70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 --2- 172.21.15.39:0/132250288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3301039b0 0x7fe33019a2e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 >> 172.21.15.39:0/132250288 conn(0x7fe3300ff440 msgr2=0x7fe330101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 shutdown_connections 2024-09-21T01:01:41.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:41.366+0000 7fe336c00640 1 -- 172.21.15.39:0/132250288 wait complete. 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 1 op/s 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/132250288' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:01:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:01:41.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 1 op/s 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/132250288' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:01:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:01:41.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 1 op/s 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/132250288' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:01:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:01:42.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs set cephfs session_timeout 300 2024-09-21T01:01:42.504 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:42.643 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:42 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T01:01:42.643 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:42 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-09-21T01:01:42.643 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:42 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:01:42.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:42 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T01:01:42.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:42 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-09-21T01:01:42.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:42 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:01:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:42 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T01:01:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:42 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-09-21T01:01:42.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:42 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 -- 172.21.15.39:0/1241049246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c109c40 msgr2=0x7f212c110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 --2- 172.21.15.39:0/1241049246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c109c40 0x7f212c110470 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2124009ca0 tx=0x7f212402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 -- 172.21.15.39:0/1241049246 shutdown_connections 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 --2- 172.21.15.39:0/1241049246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c109c40 0x7f212c110470 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 --2- 172.21.15.39:0/1241049246 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c1092f0 0x7f212c109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 --2- 172.21.15.39:0/1241049246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f212c1039b0 0x7f212c103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 -- 172.21.15.39:0/1241049246 >> 172.21.15.39:0/1241049246 conn(0x7f212c0ff440 msgr2=0x7f212c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 -- 172.21.15.39:0/1241049246 shutdown_connections 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.238+0000 7f2132600640 1 -- 172.21.15.39:0/1241049246 wait complete. 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.239+0000 7f2132600640 1 Processor -- start 2024-09-21T01:01:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 -- start start 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f212c1039b0 0x7f212c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 0x7f212c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c109c40 0x7f212c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f212c113900 con 0x7f212c1092f0 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f212c113780 con 0x7f212c1039b0 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2132600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f212c113a80 con 0x7f212c109c40 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2123e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 0x7f212c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2131200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f212c1039b0 0x7f212c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2123e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 0x7f212c19a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56414/0 (socket says 172.21.15.39:56414) 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2123e00640 1 -- 172.21.15.39:0/2350411831 learned_addr learned my addr 172.21.15.39:0/2350411831 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.240+0000 7f2131c00640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c109c40 0x7f212c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2123e00640 1 -- 172.21.15.39:0/2350411831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c109c40 msgr2=0x7f212c19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2123e00640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c109c40 0x7f212c19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2123e00640 1 -- 172.21.15.39:0/2350411831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f212c1039b0 msgr2=0x7f212c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2123e00640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f212c1039b0 0x7f212c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2123e00640 1 -- 172.21.15.39:0/2350411831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c1a0200 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2131c00640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c109c40 0x7f212c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2123e00640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 0x7f212c19a830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f211400e9c0 tx=0x7f211400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211400ccc0 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21140054d0 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.241+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212c1a04f0 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.242+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2114010430 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.242+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f212c1a6c50 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.243+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212c103dc0 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.247+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2114005000 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.248+0000 7f212a000640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21100b6560 0x7f21100b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.249+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f21140e7de0 con 0x7f212c1092f0 2024-09-21T01:01:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.249+0000 7f2131200640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21100b6560 0x7f21100b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:43.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.250+0000 7f2131200640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21100b6560 0x7f21100b8a70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f211c009bd0 tx=0x7f211c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:43.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.251+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2114056710 con 0x7f212c1092f0 2024-09-21T01:01:43.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:43.492+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f212c02e640 con 0x7f212c1092f0 2024-09-21T01:01:43.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:43 smithi039 ceph-mon[30415]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-21T01:01:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:43 smithi150 ceph-mon[34945]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-21T01:01:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:43 smithi160 ceph-mon[34696]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-21T01:01:44.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.377+0000 7f212a000640 1 -- 172.21.15.39:0/2350411831 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v26) ==== 115+0+0 (secure 0 0 0) 0x7f21140052f0 con 0x7f212c1092f0 2024-09-21T01:01:44.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21100b6560 msgr2=0x7f21100b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21100b6560 0x7f21100b8a70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f211c009bd0 tx=0x7f211c005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 msgr2=0x7f212c19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 0x7f212c19a830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f211400e9c0 tx=0x7f211400ee80 comp rx=0 tx=0).stop 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 shutdown_connections 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f212c109c40 0x7f212c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c1092f0 0x7f212c19a830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21100b6560 0x7f21100b8a70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 --2- 172.21.15.39:0/2350411831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f212c1039b0 0x7f212c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 >> 172.21.15.39:0/2350411831 conn(0x7f212c0ff440 msgr2=0x7f212c101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.380+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 shutdown_connections 2024-09-21T01:01:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:44.381+0000 7f2132600640 1 -- 172.21.15.39:0/2350411831 wait complete. 2024-09-21T01:01:44.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:44 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2350411831' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T01:01:44.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:44 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2350411831' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T01:01:44.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:44 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2350411831' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T01:01:45.189 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-09-21T01:01:45.189 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs subvolumegroup create cephfs qa 2024-09-21T01:01:45.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:45 smithi039 ceph-mon[30415]: pgmap v334: 129 pgs: 129 active+clean; 584 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-21T01:01:45.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2350411831' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T01:01:45.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:45 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:01:45.574 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:45 smithi150 ceph-mon[34945]: pgmap v334: 129 pgs: 129 active+clean; 584 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-21T01:01:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2350411831' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T01:01:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:45 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:01:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:45 smithi160 ceph-mon[34696]: pgmap v334: 129 pgs: 129 active+clean; 584 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-09-21T01:01:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2350411831' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T01:01:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:45 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 -- 172.21.15.39:0/3964207254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 msgr2=0x7fda141098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 --2- 172.21.15.39:0/3964207254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda141098b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009900 tx=0x7fd9fc02a410 comp rx=0 tx=0).stop 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 -- 172.21.15.39:0/3964207254 shutdown_connections 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 --2- 172.21.15.39:0/3964207254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14109df0 0x7fda14110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 --2- 172.21.15.39:0/3964207254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda141098b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 --2- 172.21.15.39:0/3964207254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14103b60 0x7fda14103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 -- 172.21.15.39:0/3964207254 >> 172.21.15.39:0/3964207254 conn(0x7fda1407b850 msgr2=0x7fda1407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:46.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 -- 172.21.15.39:0/3964207254 shutdown_connections 2024-09-21T01:01:46.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.274+0000 7fda19200640 1 -- 172.21.15.39:0/3964207254 wait complete. 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 Processor -- start 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 -- start start 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14103b60 0x7fda1419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda1419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14109df0 0x7fda141a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fda14113ab0 con 0x7fda141094a0 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fda14113930 con 0x7fda14103b60 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda19200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fda14113c30 con 0x7fda14109df0 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.276+0000 7fda09400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda1419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda1419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56432/0 (socket says 172.21.15.39:56432) 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda11e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14103b60 0x7fda1419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 -- 172.21.15.39:0/251944222 learned_addr learned my addr 172.21.15.39:0/251944222 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:46.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda12800640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14109df0 0x7fda141a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:46.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 -- 172.21.15.39:0/251944222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14109df0 msgr2=0x7fda141a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14109df0 0x7fda141a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 -- 172.21.15.39:0/251944222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14103b60 msgr2=0x7fda1419c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14103b60 0x7fda1419c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda09400640 1 -- 172.21.15.39:0/251944222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda141a26b0 con 0x7fda141094a0 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda12800640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14109df0 0x7fda141a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.277+0000 7fda11e00640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14103b60 0x7fda1419c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.278+0000 7fda09400640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda1419cce0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc02a930 tx=0x7fd9fc031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.278+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc03f030 con 0x7fda141094a0 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.278+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9fc004490 con 0x7fda141094a0 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.278+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda141a2940 con 0x7fda141094a0 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.278+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc035070 con 0x7fda141094a0 2024-09-21T01:01:46.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.278+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda141a9100 con 0x7fda141094a0 2024-09-21T01:01:46.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.279+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda14104de0 con 0x7fda141094a0 2024-09-21T01:01:46.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.281+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fd9fc042030 con 0x7fda141094a0 2024-09-21T01:01:46.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.283+0000 7fda0b400640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9f80b6560 0x7fd9f80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:46.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.283+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fd9fc107ed0 con 0x7fda141094a0 2024-09-21T01:01:46.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.283+0000 7fda11e00640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9f80b6560 0x7fd9f80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:46.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.284+0000 7fda11e00640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9f80b6560 0x7fd9f80b8a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fda04009fc0 tx=0x7fda04005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:46.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.286+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd9fc0d26b0 con 0x7fda141094a0 2024-09-21T01:01:46.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.517+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fda14105a00 con 0x7fd9f80b6560 2024-09-21T01:01:46.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.551+0000 7fda0b400640 1 -- 172.21.15.39:0/251944222 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fda14105a00 con 0x7fd9f80b6560 2024-09-21T01:01:46.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9f80b6560 msgr2=0x7fd9f80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:46.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9f80b6560 0x7fd9f80b8a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fda04009fc0 tx=0x7fda04005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 msgr2=0x7fda1419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda1419cce0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc02a930 tx=0x7fd9fc031a20 comp rx=0 tx=0).stop 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 shutdown_connections 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fda14109df0 0x7fda141a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda141094a0 0x7fda1419cce0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9f80b6560 0x7fd9f80b8a70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 --2- 172.21.15.39:0/251944222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda14103b60 0x7fda1419c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 >> 172.21.15.39:0/251944222 conn(0x7fda1407b850 msgr2=0x7fda141052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 shutdown_connections 2024-09-21T01:01:46.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:46.554+0000 7fda19200640 1 -- 172.21.15.39:0/251944222 wait complete. 2024-09-21T01:01:46.903 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:01:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:01:46.524+0000 7f4daa000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:01:46.904 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:01:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:01:46.524+0000 7f4daa000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:01:46.904 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:01:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:01:46.524+0000 7f4daa000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:01:46.904 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:01:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:01:46.524+0000 7f4daa000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:01:46.904 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:01:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:01:46.524+0000 7f4daa000640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:01:47.515 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:47 smithi039 ceph-mon[30415]: pgmap v335: 129 pgs: 129 active+clean; 584 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T01:01:47.515 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:47 smithi039 ceph-mon[30415]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:47.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-21T01:01:47.659 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:47 smithi160 ceph-mon[34696]: pgmap v335: 129 pgs: 129 active+clean; 584 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T01:01:47.738 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:47 smithi160 ceph-mon[34696]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:47 smithi150 ceph-mon[34945]: pgmap v335: 129 pgs: 129 active+clean; 584 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T01:01:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:47 smithi150 ceph-mon[34945]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:47.905 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 -- 172.21.15.39:0/1405776915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10be40 msgr2=0x7fed4c112670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 --2- 172.21.15.39:0/1405776915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10be40 0x7fed4c112670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed44009ca0 tx=0x7fed4402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 -- 172.21.15.39:0/1405776915 shutdown_connections 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 --2- 172.21.15.39:0/1405776915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10be40 0x7fed4c112670 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 --2- 172.21.15.39:0/1405776915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed4c10b4f0 0x7fed4c10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 --2- 172.21.15.39:0/1405776915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4c105bb0 0x7fed4c105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 -- 172.21.15.39:0/1405776915 >> 172.21.15.39:0/1405776915 conn(0x7fed4c101680 msgr2=0x7fed4c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 -- 172.21.15.39:0/1405776915 shutdown_connections 2024-09-21T01:01:48.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.654+0000 7fed51600640 1 -- 172.21.15.39:0/1405776915 wait complete. 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.655+0000 7fed51600640 1 Processor -- start 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.655+0000 7fed51600640 1 -- start start 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed51600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4c105bb0 0x7fed4c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed51600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 0x7fed4c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed51600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed4c10be40 0x7fed4c1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed51600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c115b50 con 0x7fed4c10b4f0 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed51600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1159d0 con 0x7fed4c105bb0 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed51600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c115cd0 con 0x7fed4c10be40 2024-09-21T01:01:48.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 0x7fed4c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 0x7fed4c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56478/0 (socket says 172.21.15.39:56478) 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 -- 172.21.15.39:0/2752438811 learned_addr learned my addr 172.21.15.39:0/2752438811 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed4be00640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4c105bb0 0x7fed4c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed50c00640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed4c10be40 0x7fed4c1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 -- 172.21.15.39:0/2752438811 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed4c10be40 msgr2=0x7fed4c1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed4c10be40 0x7fed4c1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 -- 172.21.15.39:0/2752438811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4c105bb0 msgr2=0x7fed4c19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4c105bb0 0x7fed4c19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.656+0000 7fed43400640 1 -- 172.21.15.39:0/2752438811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c1a23f0 con 0x7fed4c10b4f0 2024-09-21T01:01:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.657+0000 7fed43400640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 0x7fed4c19ca20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed3400e9f0 tx=0x7fed3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:48.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.657+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3400ccc0 con 0x7fed4c10b4f0 2024-09-21T01:01:48.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.657+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed340108e0 con 0x7fed4c10b4f0 2024-09-21T01:01:48.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.657+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c1a26e0 con 0x7fed4c10b4f0 2024-09-21T01:01:48.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.657+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34018860 con 0x7fed4c10b4f0 2024-09-21T01:01:48.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.657+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c1a8e40 con 0x7fed4c10b4f0 2024-09-21T01:01:48.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.658+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4c105fc0 con 0x7fed4c10b4f0 2024-09-21T01:01:48.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.663+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fed34010430 con 0x7fed4c10b4f0 2024-09-21T01:01:48.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.664+0000 7fed49600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed300b6670 0x7fed300b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:48.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.664+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fed340e8080 con 0x7fed4c10b4f0 2024-09-21T01:01:48.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.664+0000 7fed4be00640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed300b6670 0x7fed300b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:48.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.665+0000 7fed4be00640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed300b6670 0x7fed300b8b80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009bd0 tx=0x7fed3c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:48.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.666+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fed340b2700 con 0x7fed4c10b4f0 2024-09-21T01:01:48.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.891+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fed4c063800 con 0x7fed300b6670 2024-09-21T01:01:48.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.922+0000 7fed49600640 1 -- 172.21.15.39:0/2752438811 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fed4c063800 con 0x7fed300b6670 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed300b6670 msgr2=0x7fed300b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed300b6670 0x7fed300b8b80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009bd0 tx=0x7fed3c005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 msgr2=0x7fed4c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 0x7fed4c19ca20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed3400e9f0 tx=0x7fed3400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 shutdown_connections 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fed4c10be40 0x7fed4c1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed4c10b4f0 0x7fed4c19ca20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fed300b6670 0x7fed300b8b80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 --2- 172.21.15.39:0/2752438811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4c105bb0 0x7fed4c19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.924+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 >> 172.21.15.39:0/2752438811 conn(0x7fed4c101680 msgr2=0x7fed4c10aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.925+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 shutdown_connections 2024-09-21T01:01:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:48.925+0000 7fed51600640 1 -- 172.21.15.39:0/2752438811 wait complete. 2024-09-21T01:01:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:49 smithi039 ceph-mon[30415]: pgmap v336: 129 pgs: 129 active+clean; 585 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-09-21T01:01:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:49 smithi039 ceph-mon[30415]: mgrmap e21: x(active, since 8m), standbys: z, y 2024-09-21T01:01:49.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-21T01:01:49.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:49 smithi150 ceph-mon[34945]: pgmap v336: 129 pgs: 129 active+clean; 585 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-09-21T01:01:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:49 smithi150 ceph-mon[34945]: mgrmap e21: x(active, since 8m), standbys: z, y 2024-09-21T01:01:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:49 smithi160 ceph-mon[34696]: pgmap v336: 129 pgs: 129 active+clean; 585 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-09-21T01:01:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:49 smithi160 ceph-mon[34696]: mgrmap e21: x(active, since 8m), standbys: z, y 2024-09-21T01:01:50.185 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:50.631 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:50 smithi039 ceph-mon[30415]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:50 smithi150 ceph-mon[34945]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:50.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 -- 172.21.15.39:0/2499956169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 msgr2=0x7faa24103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:50.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 --2- 172.21.15.39:0/2499956169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24103dc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa14009900 tx=0x7faa1402a410 comp rx=0 tx=0).stop 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 -- 172.21.15.39:0/2499956169 shutdown_connections 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 --2- 172.21.15.39:0/2499956169 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faa24109c40 0x7faa24110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 --2- 172.21.15.39:0/2499956169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa241092f0 0x7faa24109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 --2- 172.21.15.39:0/2499956169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24103dc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 -- 172.21.15.39:0/2499956169 >> 172.21.15.39:0/2499956169 conn(0x7faa240ff440 msgr2=0x7faa24101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 -- 172.21.15.39:0/2499956169 shutdown_connections 2024-09-21T01:01:50.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.816+0000 7faa23200640 1 -- 172.21.15.39:0/2499956169 wait complete. 2024-09-21T01:01:50.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.818+0000 7faa23200640 1 Processor -- start 2024-09-21T01:01:50.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.818+0000 7faa23200640 1 -- start start 2024-09-21T01:01:50.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.818+0000 7faa23200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:50.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.818+0000 7faa23200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faa241092f0 0x7faa24075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:50.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.818+0000 7faa23200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24109c40 0x7faa24076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:50.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa23200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faa24113950 con 0x7faa241039b0 2024-09-21T01:01:50.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa23200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faa241137d0 con 0x7faa24109c40 2024-09-21T01:01:50.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa23200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faa24113ad0 con 0x7faa241092f0 2024-09-21T01:01:50.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:50.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56492/0 (socket says 172.21.15.39:56492) 2024-09-21T01:01:50.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 -- 172.21.15.39:0/36666146 learned_addr learned my addr 172.21.15.39:0/36666146 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:50.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa1be00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faa241092f0 0x7faa24075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:50.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa22800640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24109c40 0x7faa24076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 -- 172.21.15.39:0/36666146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faa241092f0 msgr2=0x7faa24075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faa241092f0 0x7faa24075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 -- 172.21.15.39:0/36666146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24109c40 msgr2=0x7faa24076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24109c40 0x7faa24076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.819+0000 7faa21e00640 1 -- 172.21.15.39:0/36666146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241a85e0 con 0x7faa241039b0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.820+0000 7faa21e00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24075710 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa14005ab0 tx=0x7faa14003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.820+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1403f070 con 0x7faa241039b0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.820+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa14031b60 con 0x7faa241039b0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.820+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa241a8870 con 0x7faa241039b0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.820+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa14035030 con 0x7faa241039b0 2024-09-21T01:01:50.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.820+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa241a8d50 con 0x7faa241039b0 2024-09-21T01:01:50.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.821+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa24103dc0 con 0x7faa241039b0 2024-09-21T01:01:50.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.824+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7faa14042070 con 0x7faa241039b0 2024-09-21T01:01:50.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.825+0000 7faa1aa00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faa080b65f0 0x7faa080b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:50.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.825+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7faa14108070 con 0x7faa241039b0 2024-09-21T01:01:50.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.825+0000 7faa1be00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faa080b65f0 0x7faa080b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:50.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.826+0000 7faa1be00640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faa080b65f0 0x7faa080b8b00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faa24108df0 tx=0x7faa0c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:50.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:50.829+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faa140d26f0 con 0x7faa241039b0 2024-09-21T01:01:50.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:50 smithi160 ceph-mon[34696]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:51.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.055+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7faa24063800 con 0x7faa080b65f0 2024-09-21T01:01:51.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.063+0000 7faa1aa00640 1 -- 172.21.15.39:0/36666146 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7faa24063800 con 0x7faa080b65f0 2024-09-21T01:01:51.064 INFO:teuthology.orchestra.run.smithi039.stdout:/volumes/qa/sv_0/4411d9bd-bd8f-4a55-a388-2fc415daa05c 2024-09-21T01:01:51.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faa080b65f0 msgr2=0x7faa080b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faa080b65f0 0x7faa080b8b00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faa24108df0 tx=0x7faa0c005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 msgr2=0x7faa24075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24075710 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa14005ab0 tx=0x7faa14003c10 comp rx=0 tx=0).stop 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 shutdown_connections 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faa080b65f0 0x7faa080b8b00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24109c40 0x7faa24076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faa241092f0 0x7faa24075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 --2- 172.21.15.39:0/36666146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa241039b0 0x7faa24075710 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.065+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 >> 172.21.15.39:0/36666146 conn(0x7faa240ff440 msgr2=0x7faa24101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.066+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 shutdown_connections 2024-09-21T01:01:51.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:51.066+0000 7faa23200640 1 -- 172.21.15.39:0/36666146 wait complete. 2024-09-21T01:01:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:51 smithi150 ceph-mon[34945]: pgmap v337: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.9 KiB/s wr, 13 op/s 2024-09-21T01:01:51.847 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:51 smithi039 ceph-mon[30415]: pgmap v337: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.9 KiB/s wr, 13 op/s 2024-09-21T01:01:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:51 smithi160 ceph-mon[34696]: pgmap v337: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.9 KiB/s wr, 13 op/s 2024-09-21T01:01:51.886 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-21T01:01:52.266 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:52.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:52 smithi039 ceph-mon[30415]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:52 smithi150 ceph-mon[34945]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:52 smithi160 ceph-mon[34696]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:52.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 -- 172.21.15.39:0/1002806661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f40757b0 msgr2=0x7f63f4075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:52.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 --2- 172.21.15.39:0/1002806661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f40757b0 0x7f63f4075bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009900 tx=0x7f63e802a410 comp rx=0 tx=0).stop 2024-09-21T01:01:52.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 -- 172.21.15.39:0/1002806661 shutdown_connections 2024-09-21T01:01:52.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 --2- 172.21.15.39:0/1002806661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f41100e0 0x7f63f4112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:52.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 --2- 172.21.15.39:0/1002806661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f63f4076100 0x7f63f410fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:52.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 --2- 172.21.15.39:0/1002806661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f40757b0 0x7f63f4075bc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:52.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 -- 172.21.15.39:0/1002806661 >> 172.21.15.39:0/1002806661 conn(0x7f63f40ff5d0 msgr2=0x7f63f4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:52.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 -- 172.21.15.39:0/1002806661 shutdown_connections 2024-09-21T01:01:52.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.867+0000 7f63fbc00640 1 -- 172.21.15.39:0/1002806661 wait complete. 2024-09-21T01:01:52.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 Processor -- start 2024-09-21T01:01:52.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 -- start start 2024-09-21T01:01:52.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f40757b0 0x7f63f419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:52.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f63f4076100 0x7f63f419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 0x7f63f41a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4115c90 con 0x7f63f41100e0 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4115b10 con 0x7f63f40757b0 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.868+0000 7f63fbc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4115e10 con 0x7f63f4076100 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 0x7f63f41a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f63f4076100 0x7f63f419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 0x7f63f41a1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54832/0 (socket says 172.21.15.39:54832) 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 -- 172.21.15.39:0/3448334652 learned_addr learned my addr 172.21.15.39:0/3448334652 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9200640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f40757b0 0x7f63f419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:52.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 -- 172.21.15.39:0/3448334652 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f63f4076100 msgr2=0x7f63f419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:52.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f63f4076100 0x7f63f419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:52.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 -- 172.21.15.39:0/3448334652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f40757b0 msgr2=0x7f63f419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:52.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f40757b0 0x7f63f419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:52.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.869+0000 7f63f9c00640 1 -- 172.21.15.39:0/3448334652 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f41a2630 con 0x7f63f41100e0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.870+0000 7f63f9c00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 0x7f63f41a1f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00eb10 tx=0x7f63ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.870+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec004b20 con 0x7f63f41100e0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.870+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f41a2920 con 0x7f63f41100e0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.870+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63ec004cc0 con 0x7f63f41100e0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.870+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec010670 con 0x7f63f41100e0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.870+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f41a9080 con 0x7f63f41100e0 2024-09-21T01:01:52.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.871+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63f4075bc0 con 0x7f63f41100e0 2024-09-21T01:01:52.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.873+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f63ec005000 con 0x7f63f41100e0 2024-09-21T01:01:52.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.875+0000 7f63f1600640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f63dc0b65f0 0x7f63dc0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:52.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.875+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f63ec0e80a0 con 0x7f63f41100e0 2024-09-21T01:01:52.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.875+0000 7f63f9200640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f63dc0b65f0 0x7f63dc0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:52.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.876+0000 7f63f9200640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f63dc0b65f0 0x7f63dc0b8b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63e8031440 tx=0x7f63e8009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:52.877+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f63ec056710 con 0x7f63f41100e0 2024-09-21T01:01:53.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.107+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f63f410a390 con 0x7f63dc0b65f0 2024-09-21T01:01:53.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.135+0000 7f63f1600640 1 -- 172.21.15.39:0/3448334652 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f63f410a390 con 0x7f63dc0b65f0 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.137+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f63dc0b65f0 msgr2=0x7f63dc0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.137+0000 7f63fbc00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f63dc0b65f0 0x7f63dc0b8b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63e8031440 tx=0x7f63e8009560 comp rx=0 tx=0).stop 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.137+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 msgr2=0x7f63f41a1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.137+0000 7f63fbc00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 0x7f63f41a1f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00eb10 tx=0x7f63ec00c2d0 comp rx=0 tx=0).stop 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 shutdown_connections 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63f41100e0 0x7f63f41a1f00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f63f4076100 0x7f63f419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:53.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f63dc0b65f0 0x7f63dc0b8b00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:53.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 --2- 172.21.15.39:0/3448334652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f40757b0 0x7f63f419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:53.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 >> 172.21.15.39:0/3448334652 conn(0x7f63f40ff5d0 msgr2=0x7f63f4109c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:53.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 shutdown_connections 2024-09-21T01:01:53.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:53.138+0000 7f63fbc00640 1 -- 172.21.15.39:0/3448334652 wait complete. 2024-09-21T01:01:53.674 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:53 smithi039 ceph-mon[30415]: pgmap v338: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-09-21T01:01:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:53 smithi150 ceph-mon[34945]: pgmap v338: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-09-21T01:01:53.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:53 smithi160 ceph-mon[34696]: pgmap v338: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-09-21T01:01:54.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-21T01:01:54.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:54 smithi039 ceph-mon[30415]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:54.720 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:54.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:54 smithi150 ceph-mon[34945]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:54.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:54 smithi160 ceph-mon[34696]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 -- 172.21.15.39:0/1542789209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 msgr2=0x7f7364112680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 --2- 172.21.15.39:0/1542789209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364112680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f735c009ca0 tx=0x7f735c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 -- 172.21.15.39:0/1542789209 shutdown_connections 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 --2- 172.21.15.39:0/1542789209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364112680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 --2- 172.21.15.39:0/1542789209 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 0x7f736410b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 --2- 172.21.15.39:0/1542789209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f7364105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 -- 172.21.15.39:0/1542789209 >> 172.21.15.39:0/1542789209 conn(0x7f7364101630 msgr2=0x7f7364103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 -- 172.21.15.39:0/1542789209 shutdown_connections 2024-09-21T01:01:55.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.343+0000 7f7368e00640 1 -- 172.21.15.39:0/1542789209 wait complete. 2024-09-21T01:01:55.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.344+0000 7f7368e00640 1 Processor -- start 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.344+0000 7f7368e00640 1 -- start start 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7368e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f736407b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7368e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 0x7f7364077920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7368e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364078040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7368e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7364115b60 con 0x7f736410be50 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7368e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f73641159e0 con 0x7f7364105bc0 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7368e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7364115ce0 con 0x7f736410b500 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7363400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f736407b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7363400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f736407b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:49562/0 (socket says 172.21.15.39:49562) 2024-09-21T01:01:55.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f735aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 0x7f7364077920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7363400640 1 -- 172.21.15.39:0/3842733523 learned_addr learned my addr 172.21.15.39:0/3842733523 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.345+0000 7f7363e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364078040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363400640 1 -- 172.21.15.39:0/3842733523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 msgr2=0x7f7364077920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363400640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 0x7f7364077920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363400640 1 -- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 msgr2=0x7f7364078040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363400640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364078040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363400640 1 -- 172.21.15.39:0/3842733523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73641aca20 con 0x7f7364105bc0 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f735aa00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 0x7f7364077920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:55.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364078040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7363400640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f736407b640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f735400e9f0 tx=0x7f735400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735400cd80 con 0x7f7364105bc0 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73540108e0 con 0x7f7364105bc0 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73641acc50 con 0x7f7364105bc0 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354018980 con 0x7f7364105bc0 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.346+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73641ad190 con 0x7f7364105bc0 2024-09-21T01:01:55.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.347+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7364105fd0 con 0x7f7364105bc0 2024-09-21T01:01:55.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.350+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f7354010430 con 0x7f7364105bc0 2024-09-21T01:01:55.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.351+0000 7f7360c00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73480b65f0 0x7f73480b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:55.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.352+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f73540e81a0 con 0x7f7364105bc0 2024-09-21T01:01:55.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.352+0000 7f735aa00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73480b65f0 0x7f73480b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:55.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.352+0000 7f735aa00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73480b65f0 0x7f73480b8b00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7364104d30 tx=0x7f734c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:55.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.354+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f73540ae4a0 con 0x7f7364105bc0 2024-09-21T01:01:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.583+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f7364063800 con 0x7f73480b65f0 2024-09-21T01:01:55.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.590+0000 7f7360c00640 1 -- 172.21.15.39:0/3842733523 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7364063800 con 0x7f73480b65f0 2024-09-21T01:01:55.591 INFO:teuthology.orchestra.run.smithi039.stdout:/volumes/qa/sv_1/c561b996-0695-4404-89f5-09c45de940bd 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73480b65f0 msgr2=0x7f73480b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73480b65f0 0x7f73480b8b00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7364104d30 tx=0x7f734c005f90 comp rx=0 tx=0).stop 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 msgr2=0x7f736407b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f736407b640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f735400e9f0 tx=0x7f735400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 shutdown_connections 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f736410be50 0x7f7364078040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f736410b500 0x7f7364077920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f73480b65f0 0x7f73480b8b00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 --2- 172.21.15.39:0/3842733523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7364105bc0 0x7f736407b640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 >> 172.21.15.39:0/3842733523 conn(0x7f7364101630 msgr2=0x7f736410ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.592+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 shutdown_connections 2024-09-21T01:01:55.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:55.593+0000 7f7368e00640 1 -- 172.21.15.39:0/3842733523 wait complete. 2024-09-21T01:01:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:55 smithi150 ceph-mon[34945]: pgmap v339: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-21T01:01:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:55 smithi160 ceph-mon[34696]: pgmap v339: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-21T01:01:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:55 smithi039 ceph-mon[30415]: pgmap v339: 129 pgs: 129 active+clean; 611 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-21T01:01:56.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:01:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:56 smithi039 ceph-mon[30415]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:56.813 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:56.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:56 smithi150 ceph-mon[34945]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:56.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:56 smithi160 ceph-mon[34696]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 -- 172.21.15.39:0/2805492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6810b4f0 msgr2=0x7faf6810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 --2- 172.21.15.39:0/2805492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6810b4f0 0x7faf6810b900 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faf50009900 tx=0x7faf5002a410 comp rx=0 tx=0).stop 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 -- 172.21.15.39:0/2805492 shutdown_connections 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 --2- 172.21.15.39:0/2805492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf6810be40 0x7faf68112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 --2- 172.21.15.39:0/2805492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6810b4f0 0x7faf6810b900 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 --2- 172.21.15.39:0/2805492 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf68105bb0 0x7faf68105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 -- 172.21.15.39:0/2805492 >> 172.21.15.39:0/2805492 conn(0x7faf68101680 msgr2=0x7faf68103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:57.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 -- 172.21.15.39:0/2805492 shutdown_connections 2024-09-21T01:01:57.510 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:57 smithi039 ceph-mon[30415]: pgmap v340: 129 pgs: 129 active+clean; 636 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.3 KiB/s wr, 9 op/s 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.508+0000 7faf6ca00640 1 -- 172.21.15.39:0/2805492 wait complete. 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.509+0000 7faf6ca00640 1 Processor -- start 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.509+0000 7faf6ca00640 1 -- start start 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf6ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 0x7faf6819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf6ca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf6810b4f0 0x7faf6819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf6ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf6810be40 0x7faf681a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf6ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faf68115b00 con 0x7faf68105bb0 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf6ca00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faf68115980 con 0x7faf6810b4f0 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf6ca00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faf68115c80 con 0x7faf6810be40 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf66800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 0x7faf6819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf66800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 0x7faf6819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54874/0 (socket says 172.21.15.39:54874) 2024-09-21T01:01:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf66800640 1 -- 172.21.15.39:0/1951094085 learned_addr learned my addr 172.21.15.39:0/1951094085 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf67200640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf6810be40 0x7faf681a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.510+0000 7faf5fe00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf6810b4f0 0x7faf6819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf66800640 1 -- 172.21.15.39:0/1951094085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf6810be40 msgr2=0x7faf681a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf66800640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf6810be40 0x7faf681a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf66800640 1 -- 172.21.15.39:0/1951094085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf6810b4f0 msgr2=0x7faf6819cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf66800640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf6810b4f0 0x7faf6819cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf66800640 1 -- 172.21.15.39:0/1951094085 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf681a2480 con 0x7faf68105bb0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf66800640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 0x7faf6819c510 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faf5800e9f0 tx=0x7faf5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.511+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5800cd80 con 0x7faf68105bb0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.512+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf681a2770 con 0x7faf68105bb0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.512+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf580054d0 con 0x7faf68105bb0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.512+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58010690 con 0x7faf68105bb0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.512+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf681a8ed0 con 0x7faf68105bb0 2024-09-21T01:01:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.513+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf68106e30 con 0x7faf68105bb0 2024-09-21T01:01:57.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.515+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7faf58004e90 con 0x7faf68105bb0 2024-09-21T01:01:57.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.517+0000 7faf5f400640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf4c0b65f0 0x7faf4c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:57.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.517+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7faf580e7d60 con 0x7faf68105bb0 2024-09-21T01:01:57.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.517+0000 7faf5fe00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf4c0b65f0 0x7faf4c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.518+0000 7faf5fe00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf4c0b65f0 0x7faf4c0b8b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf50009230 tx=0x7faf50031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:57.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.521+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faf580ae000 con 0x7faf68105bb0 2024-09-21T01:01:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.758+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf6802e640 con 0x7faf68105bb0 2024-09-21T01:01:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.759+0000 7faf5f400640 1 -- 172.21.15.39:0/1951094085 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11911 (secure 0 0 0) 0x7faf580b23e0 con 0x7faf68105bb0 2024-09-21T01:01:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:57.761 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"btime":"2024-09-21T01:01:44:370343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:01:43.492789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:01:57.761 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.761+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf4c0b65f0 msgr2=0x7faf4c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.761+0000 7faf6ca00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf4c0b65f0 0x7faf4c0b8b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf50009230 tx=0x7faf50031400 comp rx=0 tx=0).stop 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 msgr2=0x7faf6819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 0x7faf6819c510 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faf5800e9f0 tx=0x7faf5800edf0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 shutdown_connections 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf6810be40 0x7faf681a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf4c0b65f0 0x7faf4c0b8b00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf6810b4f0 0x7faf6819cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 --2- 172.21.15.39:0/1951094085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf68105bb0 0x7faf6819c510 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:57.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 >> 172.21.15.39:0/1951094085 conn(0x7faf68101680 msgr2=0x7faf68103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:57.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 shutdown_connections 2024-09-21T01:01:57.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:57.762+0000 7faf6ca00640 1 -- 172.21.15.39:0/1951094085 wait complete. 2024-09-21T01:01:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:57 smithi150 ceph-mon[34945]: pgmap v340: 129 pgs: 129 active+clean; 636 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.3 KiB/s wr, 9 op/s 2024-09-21T01:01:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:57 smithi160 ceph-mon[34696]: pgmap v340: 129 pgs: 129 active+clean; 636 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.3 KiB/s wr, 9 op/s 2024-09-21T01:01:58.584 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:58 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1951094085' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:58.623 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:01:58.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:58 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1951094085' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:58.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:58 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1951094085' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:01:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:01:59.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.651+0000 7f3fb8a00640 1 -- 172.21.15.39:0/152433925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb41039b0 msgr2=0x7f3fb4103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.651+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/152433925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb41039b0 0x7f3fb4103dc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009900 tx=0x7f3fa402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 -- 172.21.15.39:0/152433925 shutdown_connections 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/152433925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb4109c40 0x7f3fb4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/152433925 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fb41092f0 0x7f3fb4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/152433925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb41039b0 0x7f3fb4103dc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 -- 172.21.15.39:0/152433925 >> 172.21.15.39:0/152433925 conn(0x7f3fb40ff400 msgr2=0x7f3fb4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 -- 172.21.15.39:0/152433925 shutdown_connections 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.652+0000 7f3fb8a00640 1 -- 172.21.15.39:0/152433925 wait complete. 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 Processor -- start 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 -- start start 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 0x7f3fb419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fb41092f0 0x7f3fb419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb4109c40 0x7f3fb41a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4113900 con 0x7f3fb4109c40 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4113780 con 0x7f3fb41039b0 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.653+0000 7f3fb8a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4113a80 con 0x7f3fb41092f0 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb4109c40 0x7f3fb41a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 0x7f3fb419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 0x7f3fb419c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:49608/0 (socket says 172.21.15.39:49608) 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3400640 1 -- 172.21.15.39:0/3128938026 learned_addr learned my addr 172.21.15.39:0/3128938026 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fabe00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fb41092f0 0x7f3fb419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 -- 172.21.15.39:0/3128938026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fb41092f0 msgr2=0x7f3fb419cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fb41092f0 0x7f3fb419cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 -- 172.21.15.39:0/3128938026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 msgr2=0x7f3fb419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:59.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 0x7f3fb419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 -- 172.21.15.39:0/3128938026 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb41a24a0 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3400640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 0x7f3fb419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.654+0000 7f3fb3e00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb4109c40 0x7f3fb41a1d70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009fd0 tx=0x7f3fac00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.655+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00ce20 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.655+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fac005020 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.655+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac010690 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.655+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb41a2790 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.655+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb41a8ef0 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.656+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb4103dc0 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.658+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f3fac0054d0 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.660+0000 7f3fb0c00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f980b65f0 0x7f3f980b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.660+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f3fac0e7ff0 con 0x7f3fb4109c40 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.660+0000 7f3fb3400640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f980b65f0 0x7f3f980b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:01:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.661+0000 7f3fb3400640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f980b65f0 0x7f3f980b8b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4031440 tx=0x7f3fa40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:01:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.661+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3fac0ae2d0 con 0x7f3fb4109c40 2024-09-21T01:01:59.751 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:01:59 smithi039 ceph-mon[30415]: pgmap v341: 129 pgs: 129 active+clean; 636 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-09-21T01:01:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:01:59 smithi150 ceph-mon[34945]: pgmap v341: 129 pgs: 129 active+clean; 636 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-09-21T01:01:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:01:59 smithi160 ceph-mon[34696]: pgmap v341: 129 pgs: 129 active+clean; 636 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-09-21T01:01:59.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.861+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3fb419d660 con 0x7f3fb4109c40 2024-09-21T01:01:59.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.862+0000 7f3fb0c00640 1 -- 172.21.15.39:0/3128938026 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25143 (secure 0 0 0) 0x7f3fb419d660 con 0x7f3fb4109c40 2024-09-21T01:01:59.864 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:01:59.864 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:01:09.673169+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f980b65f0 msgr2=0x7f3f980b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f980b65f0 0x7f3f980b8b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4031440 tx=0x7f3fa40058d0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb4109c40 msgr2=0x7f3fb41a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb4109c40 0x7f3fb41a1d70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009fd0 tx=0x7f3fac00ef10 comp rx=0 tx=0).stop 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 shutdown_connections 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3fb4109c40 0x7f3fb41a1d70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fb41092f0 0x7f3fb419cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f980b65f0 0x7f3f980b8b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 --2- 172.21.15.39:0/3128938026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fb41039b0 0x7f3fb419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 >> 172.21.15.39:0/3128938026 conn(0x7f3fb40ff400 msgr2=0x7f3fb41089f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:01:59.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 shutdown_connections 2024-09-21T01:01:59.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:01:59.865+0000 7f3fb8a00640 1 -- 172.21.15.39:0/3128938026 wait complete. 2024-09-21T01:02:00.636 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T01:02:00.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:02:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:00 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3128938026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:00.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:00 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3128938026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:00.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:00 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3128938026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 -- 172.21.15.39:0/3358670916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 msgr2=0x7f3534109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 --2- 172.21.15.39:0/3358670916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f3534109700 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f351c009900 tx=0x7f351c02a410 comp rx=0 tx=0).stop 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 -- 172.21.15.39:0/3358670916 shutdown_connections 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 --2- 172.21.15.39:0/3358670916 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3534109c40 0x7f3534110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 --2- 172.21.15.39:0/3358670916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f3534109700 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 --2- 172.21.15.39:0/3358670916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35341039b0 0x7f3534103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 -- 172.21.15.39:0/3358670916 >> 172.21.15.39:0/3358670916 conn(0x7f35340ff440 msgr2=0x7f3534101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 -- 172.21.15.39:0/3358670916 shutdown_connections 2024-09-21T01:02:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.671+0000 7f3533e00640 1 -- 172.21.15.39:0/3358670916 wait complete. 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.672+0000 7f3533e00640 1 Processor -- start 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.672+0000 7f3533e00640 1 -- start start 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 0x7f353419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f353419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3534109c40 0x7f35341a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3534113950 con 0x7f35341092f0 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f35341137d0 con 0x7f3534109c40 2024-09-21T01:02:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3534113ad0 con 0x7f35341039b0 2024-09-21T01:02:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f352be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f353419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 0x7f353419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 0x7f353419c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:56084/0 (socket says 172.21.15.39:56084) 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f352be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f353419cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54900/0 (socket says 172.21.15.39:54900) 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 -- 172.21.15.39:0/2785689055 learned_addr learned my addr 172.21.15.39:0/2785689055 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3533400640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3534109c40 0x7f35341a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 -- 172.21.15.39:0/2785689055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3534109c40 msgr2=0x7f35341a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3534109c40 0x7f35341a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 -- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 msgr2=0x7f353419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.673+0000 7f3532a00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f353419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f3532a00640 1 -- 172.21.15.39:0/2785689055 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35341a2490 con 0x7f35341039b0 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f3533400640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3534109c40 0x7f35341a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f3532a00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 0x7f353419c520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f352400e9f0 tx=0x7f352400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352400cd80 con 0x7f35341039b0 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35341a2780 con 0x7f35341039b0 2024-09-21T01:02:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35240108e0 con 0x7f35341039b0 2024-09-21T01:02:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524018980 con 0x7f35341039b0 2024-09-21T01:02:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.674+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a8ee0 con 0x7f35341039b0 2024-09-21T01:02:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.675+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3534103dc0 con 0x7f35341039b0 2024-09-21T01:02:01.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.678+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f3524010430 con 0x7f35341039b0 2024-09-21T01:02:01.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.679+0000 7f352b400640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35180b6670 0x7f35180b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:01.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.680+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f35240e7c40 con 0x7f35341039b0 2024-09-21T01:02:01.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.680+0000 7f352be00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35180b6670 0x7f35180b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:01.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.680+0000 7f352be00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35180b6670 0x7f35180b8b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f351c02aa00 tx=0x7f351c005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:01.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.684+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f35240b22c0 con 0x7f35341039b0 2024-09-21T01:02:01.801 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:01 smithi039 ceph-mon[30415]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-09-21T01:02:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:01 smithi150 ceph-mon[34945]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-09-21T01:02:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:01 smithi160 ceph-mon[34696]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-09-21T01:02:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.880+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f353410fe30 con 0x7f35180b6670 2024-09-21T01:02:01.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.888+0000 7f352b400640 1 -- 172.21.15.39:0/2785689055 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340350 (secure 0 0 0) 0x7f353410fe30 con 0x7f35180b6670 2024-09-21T01:02:01.895 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:01.897 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:02:01.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.895+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35180b6670 msgr2=0x7f35180b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:01.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.895+0000 7f3533e00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35180b6670 0x7f35180b8b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f351c02aa00 tx=0x7f351c005a20 comp rx=0 tx=0).stop 2024-09-21T01:02:01.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.895+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 msgr2=0x7f353419c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:01.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.895+0000 7f3533e00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 0x7f353419c520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f352400e9f0 tx=0x7f352400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 shutdown_connections 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35180b6670 0x7f35180b8b80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3534109c40 0x7f35341a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35341092f0 0x7f353419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 --2- 172.21.15.39:0/2785689055 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35341039b0 0x7f353419c520 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 >> 172.21.15.39:0/2785689055 conn(0x7f35340ff440 msgr2=0x7f353410e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 shutdown_connections 2024-09-21T01:02:01.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:01.896+0000 7f3533e00640 1 -- 172.21.15.39:0/2785689055 wait complete. 2024-09-21T01:02:02.878 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-21T01:01:59.906153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655294,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":213,"num_read_kb":201,"num_write":286,"num_write_kb":2265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371448,"kb_used_data":13880,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570376,"statfs":{"total":1151940427776,"available":1151560065024,"internally_reserved":0,"allocated":14213120,"data_stored":8119457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56692,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":95,"num_write":30,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003777"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.351524+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:01:38.351524+0000","last_peered":"2024-09-21T01:01:38.351524+0000","last_clean":"2024-09-21T01:01:38.351524+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:01:38.351524+0000","last_undegraded":"2024-09-21T01:01:38.351524+0000","last_fullsized":"2024-09-21T01:01:38.351524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122696+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:01:10.122696+0000","last_peered":"2024-09-21T01:01:10.122696+0000","last_clean":"2024-09-21T01:01:10.122696+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:01:10.122696+0000","last_undegraded":"2024-09-21T01:01:10.122696+0000","last_fullsized":"2024-09-21T01:01:10.122696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124288+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:01:10.124288+0000","last_peered":"2024-09-21T01:01:10.124288+0000","last_clean":"2024-09-21T01:01:10.124288+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:01:10.124288+0000","last_undegraded":"2024-09-21T01:01:10.124288+0000","last_fullsized":"2024-09-21T01:01:10.124288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.123049+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:01:10.123049+0000","last_peered":"2024-09-21T01:01:10.123049+0000","last_clean":"2024-09-21T01:01:10.123049+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:01:10.123049+0000","last_undegraded":"2024-09-21T01:01:10.123049+0000","last_fullsized":"2024-09-21T01:01:10.123049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684578+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:01:09.684578+0000","last_peered":"2024-09-21T01:01:09.684578+0000","last_clean":"2024-09-21T01:01:09.684578+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:01:09.684578+0000","last_undegraded":"2024-09-21T01:01:09.684578+0000","last_fullsized":"2024-09-21T01:01:09.684578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683982+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:01:09.683982+0000","last_peered":"2024-09-21T01:01:09.683982+0000","last_clean":"2024-09-21T01:01:09.683982+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:01:09.683982+0000","last_undegraded":"2024-09-21T01:01:09.683982+0000","last_fullsized":"2024-09-21T01:01:09.683982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.348489+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:01:38.348489+0000","last_peered":"2024-09-21T01:01:38.348489+0000","last_clean":"2024-09-21T01:01:38.348489+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:01:38.348489+0000","last_undegraded":"2024-09-21T01:01:38.348489+0000","last_fullsized":"2024-09-21T01:01:38.348489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685333+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:01:09.685333+0000","last_peered":"2024-09-21T01:01:09.685333+0000","last_clean":"2024-09-21T01:01:09.685333+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:01:09.685333+0000","last_undegraded":"2024-09-21T01:01:09.685333+0000","last_fullsized":"2024-09-21T01:01:09.685333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684503+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:01:09.684503+0000","last_peered":"2024-09-21T01:01:09.684503+0000","last_clean":"2024-09-21T01:01:09.684503+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:01:09.684503+0000","last_undegraded":"2024-09-21T01:01:09.684503+0000","last_fullsized":"2024-09-21T01:01:09.684503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681617+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:01:09.681617+0000","last_peered":"2024-09-21T01:01:09.681617+0000","last_clean":"2024-09-21T01:01:09.681617+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:01:09.681617+0000","last_undegraded":"2024-09-21T01:01:09.681617+0000","last_fullsized":"2024-09-21T01:01:09.681617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689792+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:01:09.689792+0000","last_peered":"2024-09-21T01:01:09.689792+0000","last_clean":"2024-09-21T01:01:09.689792+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:01:09.689792+0000","last_undegraded":"2024-09-21T01:01:09.689792+0000","last_fullsized":"2024-09-21T01:01:09.689792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684263+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:01:09.684263+0000","last_peered":"2024-09-21T01:01:09.684263+0000","last_clean":"2024-09-21T01:01:09.684263+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:01:09.684263+0000","last_undegraded":"2024-09-21T01:01:09.684263+0000","last_fullsized":"2024-09-21T01:01:09.684263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677837+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:01:09.677837+0000","last_peered":"2024-09-21T01:01:09.677837+0000","last_clean":"2024-09-21T01:01:09.677837+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:01:09.677837+0000","last_undegraded":"2024-09-21T01:01:09.677837+0000","last_fullsized":"2024-09-21T01:01:09.677837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.336046+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:01:37.336046+0000","last_peered":"2024-09-21T01:01:37.336046+0000","last_clean":"2024-09-21T01:01:37.336046+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:01:37.336046+0000","last_undegraded":"2024-09-21T01:01:37.336046+0000","last_fullsized":"2024-09-21T01:01:37.336046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684905+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:01:09.684905+0000","last_peered":"2024-09-21T01:01:09.684905+0000","last_clean":"2024-09-21T01:01:09.684905+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:01:09.684905+0000","last_undegraded":"2024-09-21T01:01:09.684905+0000","last_fullsized":"2024-09-21T01:01:09.684905+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122273+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:01:10.122273+0000","last_peered":"2024-09-21T01:01:10.122273+0000","last_clean":"2024-09-21T01:01:10.122273+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:01:10.122273+0000","last_undegraded":"2024-09-21T01:01:10.122273+0000","last_fullsized":"2024-09-21T01:01:10.122273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685448+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:01:09.685448+0000","last_peered":"2024-09-21T01:01:09.685448+0000","last_clean":"2024-09-21T01:01:09.685448+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:01:09.685448+0000","last_undegraded":"2024-09-21T01:01:09.685448+0000","last_fullsized":"2024-09-21T01:01:09.685448+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684346+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:01:09.684346+0000","last_peered":"2024-09-21T01:01:09.684346+0000","last_clean":"2024-09-21T01:01:09.684346+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:01:09.684346+0000","last_undegraded":"2024-09-21T01:01:09.684346+0000","last_fullsized":"2024-09-21T01:01:09.684346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683851+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:01:09.683851+0000","last_peered":"2024-09-21T01:01:09.683851+0000","last_clean":"2024-09-21T01:01:09.683851+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:01:09.683851+0000","last_undegraded":"2024-09-21T01:01:09.683851+0000","last_fullsized":"2024-09-21T01:01:09.683851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.340567+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:01:37.340567+0000","last_peered":"2024-09-21T01:01:37.340567+0000","last_clean":"2024-09-21T01:01:37.340567+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:01:37.340567+0000","last_undegraded":"2024-09-21T01:01:37.340567+0000","last_fullsized":"2024-09-21T01:01:37.340567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.123608+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:01:10.123608+0000","last_peered":"2024-09-21T01:01:10.123608+0000","last_clean":"2024-09-21T01:01:10.123608+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:01:10.123608+0000","last_undegraded":"2024-09-21T01:01:10.123608+0000","last_fullsized":"2024-09-21T01:01:10.123608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681124+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:01:09.681124+0000","last_peered":"2024-09-21T01:01:09.681124+0000","last_clean":"2024-09-21T01:01:09.681124+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:01:09.681124+0000","last_undegraded":"2024-09-21T01:01:09.681124+0000","last_fullsized":"2024-09-21T01:01:09.681124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.339021+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:01:37.339021+0000","last_peered":"2024-09-21T01:01:37.339021+0000","last_clean":"2024-09-21T01:01:37.339021+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:01:37.339021+0000","last_undegraded":"2024-09-21T01:01:37.339021+0000","last_fullsized":"2024-09-21T01:01:37.339021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684922+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:01:09.684922+0000","last_peered":"2024-09-21T01:01:09.684922+0000","last_clean":"2024-09-21T01:01:09.684922+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:01:09.684922+0000","last_undegraded":"2024-09-21T01:01:09.684922+0000","last_fullsized":"2024-09-21T01:01:09.684922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124259+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:01:10.124259+0000","last_peered":"2024-09-21T01:01:10.124259+0000","last_clean":"2024-09-21T01:01:10.124259+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:01:10.124259+0000","last_undegraded":"2024-09-21T01:01:10.124259+0000","last_fullsized":"2024-09-21T01:01:10.124259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.338547+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:01:37.338547+0000","last_peered":"2024-09-21T01:01:37.338547+0000","last_clean":"2024-09-21T01:01:37.338547+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:01:37.338547+0000","last_undegraded":"2024-09-21T01:01:37.338547+0000","last_fullsized":"2024-09-21T01:01:37.338547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685671+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:01:09.685671+0000","last_peered":"2024-09-21T01:01:09.685671+0000","last_clean":"2024-09-21T01:01:09.685671+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:01:09.685671+0000","last_undegraded":"2024-09-21T01:01:09.685671+0000","last_fullsized":"2024-09-21T01:01:09.685671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122813+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:01:10.122813+0000","last_peered":"2024-09-21T01:01:10.122813+0000","last_clean":"2024-09-21T01:01:10.122813+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:01:10.122813+0000","last_undegraded":"2024-09-21T01:01:10.122813+0000","last_fullsized":"2024-09-21T01:01:10.122813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:55.516580+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:01:55.516580+0000","last_peered":"2024-09-21T01:01:55.516580+0000","last_clean":"2024-09-21T01:01:55.516580+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:01:55.516580+0000","last_undegraded":"2024-09-21T01:01:55.516580+0000","last_fullsized":"2024-09-21T01:01:55.516580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122152+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:01:10.122152+0000","last_peered":"2024-09-21T01:01:10.122152+0000","last_clean":"2024-09-21T01:01:10.122152+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:01:10.122152+0000","last_undegraded":"2024-09-21T01:01:10.122152+0000","last_fullsized":"2024-09-21T01:01:10.122152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681810+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:01:09.681810+0000","last_peered":"2024-09-21T01:01:09.681810+0000","last_clean":"2024-09-21T01:01:09.681810+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:01:09.681810+0000","last_undegraded":"2024-09-21T01:01:09.681810+0000","last_fullsized":"2024-09-21T01:01:09.681810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.338712+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:01:37.338712+0000","last_peered":"2024-09-21T01:01:37.338712+0000","last_clean":"2024-09-21T01:01:37.338712+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:01:37.338712+0000","last_undegraded":"2024-09-21T01:01:37.338712+0000","last_fullsized":"2024-09-21T01:01:37.338712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683672+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:01:09.683672+0000","last_peered":"2024-09-21T01:01:09.683672+0000","last_clean":"2024-09-21T01:01:09.683672+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:01:09.683672+0000","last_undegraded":"2024-09-21T01:01:09.683672+0000","last_fullsized":"2024-09-21T01:01:09.683672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925507+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:01:09.925507+0000","last_peered":"2024-09-21T01:01:09.925507+0000","last_clean":"2024-09-21T01:01:09.925507+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:01:09.925507+0000","last_undegraded":"2024-09-21T01:01:09.925507+0000","last_fullsized":"2024-09-21T01:01:09.925507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124130+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:01:10.124130+0000","last_peered":"2024-09-21T01:01:10.124130+0000","last_clean":"2024-09-21T01:01:10.124130+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:01:10.124130+0000","last_undegraded":"2024-09-21T01:01:10.124130+0000","last_fullsized":"2024-09-21T01:01:10.124130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681155+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:01:09.681155+0000","last_peered":"2024-09-21T01:01:09.681155+0000","last_clean":"2024-09-21T01:01:09.681155+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:01:09.681155+0000","last_undegraded":"2024-09-21T01:01:09.681155+0000","last_fullsized":"2024-09-21T01:01:09.681155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925284+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:01:09.925284+0000","last_peered":"2024-09-21T01:01:09.925284+0000","last_clean":"2024-09-21T01:01:09.925284+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:01:09.925284+0000","last_undegraded":"2024-09-21T01:01:09.925284+0000","last_fullsized":"2024-09-21T01:01:09.925284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678032+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:01:09.678032+0000","last_peered":"2024-09-21T01:01:09.678032+0000","last_clean":"2024-09-21T01:01:09.678032+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:01:09.678032+0000","last_undegraded":"2024-09-21T01:01:09.678032+0000","last_fullsized":"2024-09-21T01:01:09.678032+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681699+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:01:09.681699+0000","last_peered":"2024-09-21T01:01:09.681699+0000","last_clean":"2024-09-21T01:01:09.681699+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:01:09.681699+0000","last_undegraded":"2024-09-21T01:01:09.681699+0000","last_fullsized":"2024-09-21T01:01:09.681699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.691419+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:01:09.691419+0000","last_peered":"2024-09-21T01:01:09.691419+0000","last_clean":"2024-09-21T01:01:09.691419+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:01:09.691419+0000","last_undegraded":"2024-09-21T01:01:09.691419+0000","last_fullsized":"2024-09-21T01:01:09.691419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:55.401571+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:01:55.401571+0000","last_peered":"2024-09-21T01:01:55.401571+0000","last_clean":"2024-09-21T01:01:55.401571+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:01:55.401571+0000","last_undegraded":"2024-09-21T01:01:55.401571+0000","last_fullsized":"2024-09-21T01:01:55.401571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681687+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:01:09.681687+0000","last_peered":"2024-09-21T01:01:09.681687+0000","last_clean":"2024-09-21T01:01:09.681687+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:01:09.681687+0000","last_undegraded":"2024-09-21T01:01:09.681687+0000","last_fullsized":"2024-09-21T01:01:09.681687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681079+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:01:09.681079+0000","last_peered":"2024-09-21T01:01:09.681079+0000","last_clean":"2024-09-21T01:01:09.681079+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:01:09.681079+0000","last_undegraded":"2024-09-21T01:01:09.681079+0000","last_fullsized":"2024-09-21T01:01:09.681079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.348209+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:01:38.348209+0000","last_peered":"2024-09-21T01:01:38.348209+0000","last_clean":"2024-09-21T01:01:38.348209+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:01:38.348209+0000","last_undegraded":"2024-09-21T01:01:38.348209+0000","last_fullsized":"2024-09-21T01:01:38.348209+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.688633+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:01:09.688633+0000","last_peered":"2024-09-21T01:01:09.688633+0000","last_clean":"2024-09-21T01:01:09.688633+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:01:09.688633+0000","last_undegraded":"2024-09-21T01:01:09.688633+0000","last_fullsized":"2024-09-21T01:01:09.688633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122941+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:01:10.122941+0000","last_peered":"2024-09-21T01:01:10.122941+0000","last_clean":"2024-09-21T01:01:10.122941+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:01:10.122941+0000","last_undegraded":"2024-09-21T01:01:10.122941+0000","last_fullsized":"2024-09-21T01:01:10.122941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:41.370435+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:01:41.370435+0000","last_peered":"2024-09-21T01:01:41.370435+0000","last_clean":"2024-09-21T01:01:41.370435+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:01:41.370435+0000","last_undegraded":"2024-09-21T01:01:41.370435+0000","last_fullsized":"2024-09-21T01:01:41.370435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678560+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:01:09.678560+0000","last_peered":"2024-09-21T01:01:09.678560+0000","last_clean":"2024-09-21T01:01:09.678560+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:01:09.678560+0000","last_undegraded":"2024-09-21T01:01:09.678560+0000","last_fullsized":"2024-09-21T01:01:09.678560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.125560+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:01:10.125560+0000","last_peered":"2024-09-21T01:01:10.125560+0000","last_clean":"2024-09-21T01:01:10.125560+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:01:10.125560+0000","last_undegraded":"2024-09-21T01:01:10.125560+0000","last_fullsized":"2024-09-21T01:01:10.125560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678540+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:01:09.678540+0000","last_peered":"2024-09-21T01:01:09.678540+0000","last_clean":"2024-09-21T01:01:09.678540+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:01:09.678540+0000","last_undegraded":"2024-09-21T01:01:09.678540+0000","last_fullsized":"2024-09-21T01:01:09.678540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.351056+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:01:38.351056+0000","last_peered":"2024-09-21T01:01:38.351056+0000","last_clean":"2024-09-21T01:01:38.351056+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:01:38.351056+0000","last_undegraded":"2024-09-21T01:01:38.351056+0000","last_fullsized":"2024-09-21T01:01:38.351056+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684715+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:01:09.684715+0000","last_peered":"2024-09-21T01:01:09.684715+0000","last_clean":"2024-09-21T01:01:09.684715+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:01:09.684715+0000","last_undegraded":"2024-09-21T01:01:09.684715+0000","last_fullsized":"2024-09-21T01:01:09.684715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.126961+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:01:10.126961+0000","last_peered":"2024-09-21T01:01:10.126961+0000","last_clean":"2024-09-21T01:01:10.126961+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:01:10.126961+0000","last_undegraded":"2024-09-21T01:01:10.126961+0000","last_fullsized":"2024-09-21T01:01:10.126961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.690046+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:01:09.690046+0000","last_peered":"2024-09-21T01:01:09.690046+0000","last_clean":"2024-09-21T01:01:09.690046+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:01:09.690046+0000","last_undegraded":"2024-09-21T01:01:09.690046+0000","last_fullsized":"2024-09-21T01:01:09.690046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684349+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:01:09.684349+0000","last_peered":"2024-09-21T01:01:09.684349+0000","last_clean":"2024-09-21T01:01:09.684349+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:01:09.684349+0000","last_undegraded":"2024-09-21T01:01:09.684349+0000","last_fullsized":"2024-09-21T01:01:09.684349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678057+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:01:09.678057+0000","last_peered":"2024-09-21T01:01:09.678057+0000","last_clean":"2024-09-21T01:01:09.678057+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:01:09.678057+0000","last_undegraded":"2024-09-21T01:01:09.678057+0000","last_fullsized":"2024-09-21T01:01:09.678057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681594+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:01:09.681594+0000","last_peered":"2024-09-21T01:01:09.681594+0000","last_clean":"2024-09-21T01:01:09.681594+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:01:09.681594+0000","last_undegraded":"2024-09-21T01:01:09.681594+0000","last_fullsized":"2024-09-21T01:01:09.681594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689297+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:01:09.689297+0000","last_peered":"2024-09-21T01:01:09.689297+0000","last_clean":"2024-09-21T01:01:09.689297+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:01:09.689297+0000","last_undegraded":"2024-09-21T01:01:09.689297+0000","last_fullsized":"2024-09-21T01:01:09.689297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684237+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:01:09.684237+0000","last_peered":"2024-09-21T01:01:09.684237+0000","last_clean":"2024-09-21T01:01:09.684237+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:01:09.684237+0000","last_undegraded":"2024-09-21T01:01:09.684237+0000","last_fullsized":"2024-09-21T01:01:09.684237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:40.345501+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:01:40.345501+0000","last_peered":"2024-09-21T01:01:40.345501+0000","last_clean":"2024-09-21T01:01:40.345501+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:01:40.345501+0000","last_undegraded":"2024-09-21T01:01:40.345501+0000","last_fullsized":"2024-09-21T01:01:40.345501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.687731+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:01:09.687731+0000","last_peered":"2024-09-21T01:01:09.687731+0000","last_clean":"2024-09-21T01:01:09.687731+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:01:09.687731+0000","last_undegraded":"2024-09-21T01:01:09.687731+0000","last_fullsized":"2024-09-21T01:01:09.687731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.687911+0000","last_change":"2024-09-21T01:00:42.495651+0000","last_active":"2024-09-21T01:01:09.687911+0000","last_peered":"2024-09-21T01:01:09.687911+0000","last_clean":"2024-09-21T01:01:09.687911+0000","last_became_active":"2024-09-21T01:00:42.495096+0000","last_became_peered":"2024-09-21T01:00:42.495096+0000","last_unstale":"2024-09-21T01:01:09.687911+0000","last_undegraded":"2024-09-21T01:01:09.687911+0000","last_fullsized":"2024-09-21T01:01:09.687911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:19.981242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.354362+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:01:38.354362+0000","last_peered":"2024-09-21T01:01:38.354362+0000","last_clean":"2024-09-21T01:01:38.354362+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:01:38.354362+0000","last_undegraded":"2024-09-21T01:01:38.354362+0000","last_fullsized":"2024-09-21T01:01:38.354362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681482+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:01:09.681482+0000","last_peered":"2024-09-21T01:01:09.681482+0000","last_clean":"2024-09-21T01:01:09.681482+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:01:09.681482+0000","last_undegraded":"2024-09-21T01:01:09.681482+0000","last_fullsized":"2024-09-21T01:01:09.681482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677752+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:01:09.677752+0000","last_peered":"2024-09-21T01:01:09.677752+0000","last_clean":"2024-09-21T01:01:09.677752+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:01:09.677752+0000","last_undegraded":"2024-09-21T01:01:09.677752+0000","last_fullsized":"2024-09-21T01:01:09.677752+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:55.402342+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:01:55.402342+0000","last_peered":"2024-09-21T01:01:55.402342+0000","last_clean":"2024-09-21T01:01:55.402342+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:01:55.402342+0000","last_undegraded":"2024-09-21T01:01:55.402342+0000","last_fullsized":"2024-09-21T01:01:55.402342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.125687+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:01:10.125687+0000","last_peered":"2024-09-21T01:01:10.125687+0000","last_clean":"2024-09-21T01:01:10.125687+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:01:10.125687+0000","last_undegraded":"2024-09-21T01:01:10.125687+0000","last_fullsized":"2024-09-21T01:01:10.125687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677991+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:01:09.677991+0000","last_peered":"2024-09-21T01:01:09.677991+0000","last_clean":"2024-09-21T01:01:09.677991+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:01:09.677991+0000","last_undegraded":"2024-09-21T01:01:09.677991+0000","last_fullsized":"2024-09-21T01:01:09.677991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.339054+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:01:37.339054+0000","last_peered":"2024-09-21T01:01:37.339054+0000","last_clean":"2024-09-21T01:01:37.339054+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:01:37.339054+0000","last_undegraded":"2024-09-21T01:01:37.339054+0000","last_fullsized":"2024-09-21T01:01:37.339054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683501+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:01:09.683501+0000","last_peered":"2024-09-21T01:01:09.683501+0000","last_clean":"2024-09-21T01:01:09.683501+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:01:09.683501+0000","last_undegraded":"2024-09-21T01:01:09.683501+0000","last_fullsized":"2024-09-21T01:01:09.683501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681297+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:01:09.681297+0000","last_peered":"2024-09-21T01:01:09.681297+0000","last_clean":"2024-09-21T01:01:09.681297+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:01:09.681297+0000","last_undegraded":"2024-09-21T01:01:09.681297+0000","last_fullsized":"2024-09-21T01:01:09.681297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.353414+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:01:38.353414+0000","last_peered":"2024-09-21T01:01:38.353414+0000","last_clean":"2024-09-21T01:01:38.353414+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:01:38.353414+0000","last_undegraded":"2024-09-21T01:01:38.353414+0000","last_fullsized":"2024-09-21T01:01:38.353414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689729+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:01:09.689729+0000","last_peered":"2024-09-21T01:01:09.689729+0000","last_clean":"2024-09-21T01:01:09.689729+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:01:09.689729+0000","last_undegraded":"2024-09-21T01:01:09.689729+0000","last_fullsized":"2024-09-21T01:01:09.689729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681585+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:01:09.681585+0000","last_peered":"2024-09-21T01:01:09.681585+0000","last_clean":"2024-09-21T01:01:09.681585+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:01:09.681585+0000","last_undegraded":"2024-09-21T01:01:09.681585+0000","last_fullsized":"2024-09-21T01:01:09.681585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'30","reported_seq":115,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:59.410258+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:01:59.410258+0000","last_peered":"2024-09-21T01:01:59.410258+0000","last_clean":"2024-09-21T01:01:59.410258+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:01:59.410258+0000","last_undegraded":"2024-09-21T01:01:59.410258+0000","last_fullsized":"2024-09-21T01:01:59.410258+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60300,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":77,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124323+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:01:10.124323+0000","last_peered":"2024-09-21T01:01:10.124323+0000","last_clean":"2024-09-21T01:01:10.124323+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:01:10.124323+0000","last_undegraded":"2024-09-21T01:01:10.124323+0000","last_fullsized":"2024-09-21T01:01:10.124323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685501+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:01:09.685501+0000","last_peered":"2024-09-21T01:01:09.685501+0000","last_clean":"2024-09-21T01:01:09.685501+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:01:09.685501+0000","last_undegraded":"2024-09-21T01:01:09.685501+0000","last_fullsized":"2024-09-21T01:01:09.685501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.338691+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:01:37.338691+0000","last_peered":"2024-09-21T01:01:37.338691+0000","last_clean":"2024-09-21T01:01:37.338691+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:01:37.338691+0000","last_undegraded":"2024-09-21T01:01:37.338691+0000","last_fullsized":"2024-09-21T01:01:37.338691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689108+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:01:09.689108+0000","last_peered":"2024-09-21T01:01:09.689108+0000","last_clean":"2024-09-21T01:01:09.689108+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:01:09.689108+0000","last_undegraded":"2024-09-21T01:01:09.689108+0000","last_fullsized":"2024-09-21T01:01:09.689108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684603+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:01:09.684603+0000","last_peered":"2024-09-21T01:01:09.684603+0000","last_clean":"2024-09-21T01:01:09.684603+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:01:09.684603+0000","last_undegraded":"2024-09-21T01:01:09.684603+0000","last_fullsized":"2024-09-21T01:01:09.684603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.337512+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:01:37.337512+0000","last_peered":"2024-09-21T01:01:37.337512+0000","last_clean":"2024-09-21T01:01:37.337512+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:01:37.337512+0000","last_undegraded":"2024-09-21T01:01:37.337512+0000","last_fullsized":"2024-09-21T01:01:37.337512+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.121919+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:01:10.121919+0000","last_peered":"2024-09-21T01:01:10.121919+0000","last_clean":"2024-09-21T01:01:10.121919+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:01:10.121919+0000","last_undegraded":"2024-09-21T01:01:10.121919+0000","last_fullsized":"2024-09-21T01:01:10.121919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685964+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:01:09.685964+0000","last_peered":"2024-09-21T01:01:09.685964+0000","last_clean":"2024-09-21T01:01:09.685964+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:01:09.685964+0000","last_undegraded":"2024-09-21T01:01:09.685964+0000","last_fullsized":"2024-09-21T01:01:09.685964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:58.402318+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:01:58.402318+0000","last_peered":"2024-09-21T01:01:58.402318+0000","last_clean":"2024-09-21T01:01:58.402318+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:01:58.402318+0000","last_undegraded":"2024-09-21T01:01:58.402318+0000","last_fullsized":"2024-09-21T01:01:58.402318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122398+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:01:10.122398+0000","last_peered":"2024-09-21T01:01:10.122398+0000","last_clean":"2024-09-21T01:01:10.122398+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:01:10.122398+0000","last_undegraded":"2024-09-21T01:01:10.122398+0000","last_fullsized":"2024-09-21T01:01:10.122398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684337+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:01:09.684337+0000","last_peered":"2024-09-21T01:01:09.684337+0000","last_clean":"2024-09-21T01:01:09.684337+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:01:09.684337+0000","last_undegraded":"2024-09-21T01:01:09.684337+0000","last_fullsized":"2024-09-21T01:01:09.684337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.123309+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:01:10.123309+0000","last_peered":"2024-09-21T01:01:10.123309+0000","last_clean":"2024-09-21T01:01:10.123309+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:01:10.123309+0000","last_undegraded":"2024-09-21T01:01:10.123309+0000","last_fullsized":"2024-09-21T01:01:10.123309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689076+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:01:09.689076+0000","last_peered":"2024-09-21T01:01:09.689076+0000","last_clean":"2024-09-21T01:01:09.689076+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:01:09.689076+0000","last_undegraded":"2024-09-21T01:01:09.689076+0000","last_fullsized":"2024-09-21T01:01:09.689076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.691298+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:01:09.691298+0000","last_peered":"2024-09-21T01:01:09.691298+0000","last_clean":"2024-09-21T01:01:09.691298+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:01:09.691298+0000","last_undegraded":"2024-09-21T01:01:09.691298+0000","last_fullsized":"2024-09-21T01:01:09.691298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122088+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:01:10.122088+0000","last_peered":"2024-09-21T01:01:10.122088+0000","last_clean":"2024-09-21T01:01:10.122088+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:01:10.122088+0000","last_undegraded":"2024-09-21T01:01:10.122088+0000","last_fullsized":"2024-09-21T01:01:10.122088+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.352664+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:01:38.352664+0000","last_peered":"2024-09-21T01:01:38.352664+0000","last_clean":"2024-09-21T01:01:38.352664+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:01:38.352664+0000","last_undegraded":"2024-09-21T01:01:38.352664+0000","last_fullsized":"2024-09-21T01:01:38.352664+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.351883+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:01:38.351883+0000","last_peered":"2024-09-21T01:01:38.351883+0000","last_clean":"2024-09-21T01:01:38.351883+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:01:38.351883+0000","last_undegraded":"2024-09-21T01:01:38.351883+0000","last_fullsized":"2024-09-21T01:01:38.351883+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.688239+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:01:09.688239+0000","last_peered":"2024-09-21T01:01:09.688239+0000","last_clean":"2024-09-21T01:01:09.688239+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:01:09.688239+0000","last_undegraded":"2024-09-21T01:01:09.688239+0000","last_fullsized":"2024-09-21T01:01:09.688239+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685565+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:01:09.685565+0000","last_peered":"2024-09-21T01:01:09.685565+0000","last_clean":"2024-09-21T01:01:09.685565+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:01:09.685565+0000","last_undegraded":"2024-09-21T01:01:09.685565+0000","last_fullsized":"2024-09-21T01:01:09.685565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:55.397998+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:01:55.397998+0000","last_peered":"2024-09-21T01:01:55.397998+0000","last_clean":"2024-09-21T01:01:55.397998+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:01:55.397998+0000","last_undegraded":"2024-09-21T01:01:55.397998+0000","last_fullsized":"2024-09-21T01:01:55.397998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681227+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:01:09.681227+0000","last_peered":"2024-09-21T01:01:09.681227+0000","last_clean":"2024-09-21T01:01:09.681227+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:01:09.681227+0000","last_undegraded":"2024-09-21T01:01:09.681227+0000","last_fullsized":"2024-09-21T01:01:09.681227+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.686184+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:01:09.686184+0000","last_peered":"2024-09-21T01:01:09.686184+0000","last_clean":"2024-09-21T01:01:09.686184+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:01:09.686184+0000","last_undegraded":"2024-09-21T01:01:09.686184+0000","last_fullsized":"2024-09-21T01:01:09.686184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683461+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:01:09.683461+0000","last_peered":"2024-09-21T01:01:09.683461+0000","last_clean":"2024-09-21T01:01:09.683461+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:01:09.683461+0000","last_undegraded":"2024-09-21T01:01:09.683461+0000","last_fullsized":"2024-09-21T01:01:09.683461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685864+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:01:09.685864+0000","last_peered":"2024-09-21T01:01:09.685864+0000","last_clean":"2024-09-21T01:01:09.685864+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:01:09.685864+0000","last_undegraded":"2024-09-21T01:01:09.685864+0000","last_fullsized":"2024-09-21T01:01:09.685864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685275+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:01:09.685275+0000","last_peered":"2024-09-21T01:01:09.685275+0000","last_clean":"2024-09-21T01:01:09.685275+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:01:09.685275+0000","last_undegraded":"2024-09-21T01:01:09.685275+0000","last_fullsized":"2024-09-21T01:01:09.685275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684461+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:01:09.684461+0000","last_peered":"2024-09-21T01:01:09.684461+0000","last_clean":"2024-09-21T01:01:09.684461+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:01:09.684461+0000","last_undegraded":"2024-09-21T01:01:09.684461+0000","last_fullsized":"2024-09-21T01:01:09.684461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681765+0000","last_change":"2024-09-21T01:00:42.500918+0000","last_active":"2024-09-21T01:01:09.681765+0000","last_peered":"2024-09-21T01:01:09.681765+0000","last_clean":"2024-09-21T01:01:09.681765+0000","last_became_active":"2024-09-21T01:00:42.497579+0000","last_became_peered":"2024-09-21T01:00:42.497579+0000","last_unstale":"2024-09-21T01:01:09.681765+0000","last_undegraded":"2024-09-21T01:01:09.681765+0000","last_fullsized":"2024-09-21T01:01:09.681765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:43:22.447476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684199+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:01:09.684199+0000","last_peered":"2024-09-21T01:01:09.684199+0000","last_clean":"2024-09-21T01:01:09.684199+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:01:09.684199+0000","last_undegraded":"2024-09-21T01:01:09.684199+0000","last_fullsized":"2024-09-21T01:01:09.684199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685747+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:01:09.685747+0000","last_peered":"2024-09-21T01:01:09.685747+0000","last_clean":"2024-09-21T01:01:09.685747+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:01:09.685747+0000","last_undegraded":"2024-09-21T01:01:09.685747+0000","last_fullsized":"2024-09-21T01:01:09.685747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122642+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:01:10.122642+0000","last_peered":"2024-09-21T01:01:10.122642+0000","last_clean":"2024-09-21T01:01:10.122642+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:01:10.122642+0000","last_undegraded":"2024-09-21T01:01:10.122642+0000","last_fullsized":"2024-09-21T01:01:10.122642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124744+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:01:10.124744+0000","last_peered":"2024-09-21T01:01:10.124744+0000","last_clean":"2024-09-21T01:01:10.124744+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:01:10.124744+0000","last_undegraded":"2024-09-21T01:01:10.124744+0000","last_fullsized":"2024-09-21T01:01:10.124744+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122734+0000","last_change":"2024-09-21T01:00:42.493698+0000","last_active":"2024-09-21T01:01:10.122734+0000","last_peered":"2024-09-21T01:01:10.122734+0000","last_clean":"2024-09-21T01:01:10.122734+0000","last_became_active":"2024-09-21T01:00:42.493338+0000","last_became_peered":"2024-09-21T01:00:42.493338+0000","last_unstale":"2024-09-21T01:01:10.122734+0000","last_undegraded":"2024-09-21T01:01:10.122734+0000","last_fullsized":"2024-09-21T01:01:10.122734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:03.505872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685357+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:01:09.685357+0000","last_peered":"2024-09-21T01:01:09.685357+0000","last_clean":"2024-09-21T01:01:09.685357+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:01:09.685357+0000","last_undegraded":"2024-09-21T01:01:09.685357+0000","last_fullsized":"2024-09-21T01:01:09.685357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681522+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:01:09.681522+0000","last_peered":"2024-09-21T01:01:09.681522+0000","last_clean":"2024-09-21T01:01:09.681522+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:01:09.681522+0000","last_undegraded":"2024-09-21T01:01:09.681522+0000","last_fullsized":"2024-09-21T01:01:09.681522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683546+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:01:09.683546+0000","last_peered":"2024-09-21T01:01:09.683546+0000","last_clean":"2024-09-21T01:01:09.683546+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:01:09.683546+0000","last_undegraded":"2024-09-21T01:01:09.683546+0000","last_fullsized":"2024-09-21T01:01:09.683546+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.686367+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:01:09.686367+0000","last_peered":"2024-09-21T01:01:09.686367+0000","last_clean":"2024-09-21T01:01:09.686367+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:01:09.686367+0000","last_undegraded":"2024-09-21T01:01:09.686367+0000","last_fullsized":"2024-09-21T01:01:09.686367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.125200+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:01:10.125200+0000","last_peered":"2024-09-21T01:01:10.125200+0000","last_clean":"2024-09-21T01:01:10.125200+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:01:10.125200+0000","last_undegraded":"2024-09-21T01:01:10.125200+0000","last_fullsized":"2024-09-21T01:01:10.125200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685144+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:01:09.685144+0000","last_peered":"2024-09-21T01:01:09.685144+0000","last_clean":"2024-09-21T01:01:09.685144+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:01:09.685144+0000","last_undegraded":"2024-09-21T01:01:09.685144+0000","last_fullsized":"2024-09-21T01:01:09.685144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684137+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:01:09.684137+0000","last_peered":"2024-09-21T01:01:09.684137+0000","last_clean":"2024-09-21T01:01:09.684137+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:01:09.684137+0000","last_undegraded":"2024-09-21T01:01:09.684137+0000","last_fullsized":"2024-09-21T01:01:09.684137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684072+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:01:09.684072+0000","last_peered":"2024-09-21T01:01:09.684072+0000","last_clean":"2024-09-21T01:01:09.684072+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:01:09.684072+0000","last_undegraded":"2024-09-21T01:01:09.684072+0000","last_fullsized":"2024-09-21T01:01:09.684072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684180+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:01:09.684180+0000","last_peered":"2024-09-21T01:01:09.684180+0000","last_clean":"2024-09-21T01:01:09.684180+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:01:09.684180+0000","last_undegraded":"2024-09-21T01:01:09.684180+0000","last_fullsized":"2024-09-21T01:01:09.684180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681482+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:01:09.681482+0000","last_peered":"2024-09-21T01:01:09.681482+0000","last_clean":"2024-09-21T01:01:09.681482+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:01:09.681482+0000","last_undegraded":"2024-09-21T01:01:09.681482+0000","last_fullsized":"2024-09-21T01:01:09.681482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684764+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:01:09.684764+0000","last_peered":"2024-09-21T01:01:09.684764+0000","last_clean":"2024-09-21T01:01:09.684764+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:01:09.684764+0000","last_undegraded":"2024-09-21T01:01:09.684764+0000","last_fullsized":"2024-09-21T01:01:09.684764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678366+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:01:09.678366+0000","last_peered":"2024-09-21T01:01:09.678366+0000","last_clean":"2024-09-21T01:01:09.678366+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:01:09.678366+0000","last_undegraded":"2024-09-21T01:01:09.678366+0000","last_fullsized":"2024-09-21T01:01:09.678366+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925537+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:01:09.925537+0000","last_peered":"2024-09-21T01:01:09.925537+0000","last_clean":"2024-09-21T01:01:09.925537+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:01:09.925537+0000","last_undegraded":"2024-09-21T01:01:09.925537+0000","last_fullsized":"2024-09-21T01:01:09.925537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685903+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:01:09.685903+0000","last_peered":"2024-09-21T01:01:09.685903+0000","last_clean":"2024-09-21T01:01:09.685903+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:01:09.685903+0000","last_undegraded":"2024-09-21T01:01:09.685903+0000","last_fullsized":"2024-09-21T01:01:09.685903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.121968+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:01:10.121968+0000","last_peered":"2024-09-21T01:01:10.121968+0000","last_clean":"2024-09-21T01:01:10.121968+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:01:10.121968+0000","last_undegraded":"2024-09-21T01:01:10.121968+0000","last_fullsized":"2024-09-21T01:01:10.121968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677767+0000","last_change":"2024-09-21T01:00:42.498438+0000","last_active":"2024-09-21T01:01:09.677767+0000","last_peered":"2024-09-21T01:01:09.677767+0000","last_clean":"2024-09-21T01:01:09.677767+0000","last_became_active":"2024-09-21T01:00:42.498080+0000","last_became_peered":"2024-09-21T01:00:42.498080+0000","last_unstale":"2024-09-21T01:01:09.677767+0000","last_undegraded":"2024-09-21T01:01:09.677767+0000","last_fullsized":"2024-09-21T01:01:09.677767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:23.853505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925111+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:01:09.925111+0000","last_peered":"2024-09-21T01:01:09.925111+0000","last_clean":"2024-09-21T01:01:09.925111+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:01:09.925111+0000","last_undegraded":"2024-09-21T01:01:09.925111+0000","last_fullsized":"2024-09-21T01:01:09.925111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681265+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:01:09.681265+0000","last_peered":"2024-09-21T01:01:09.681265+0000","last_clean":"2024-09-21T01:01:09.681265+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:01:09.681265+0000","last_undegraded":"2024-09-21T01:01:09.681265+0000","last_fullsized":"2024-09-21T01:01:09.681265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684599+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:01:09.684599+0000","last_peered":"2024-09-21T01:01:09.684599+0000","last_clean":"2024-09-21T01:01:09.684599+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:01:09.684599+0000","last_undegraded":"2024-09-21T01:01:09.684599+0000","last_fullsized":"2024-09-21T01:01:09.684599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684305+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:01:09.684305+0000","last_peered":"2024-09-21T01:01:09.684305+0000","last_clean":"2024-09-21T01:01:09.684305+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:01:09.684305+0000","last_undegraded":"2024-09-21T01:01:09.684305+0000","last_fullsized":"2024-09-21T01:01:09.684305+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684461+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:01:09.684461+0000","last_peered":"2024-09-21T01:01:09.684461+0000","last_clean":"2024-09-21T01:01:09.684461+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:01:09.684461+0000","last_undegraded":"2024-09-21T01:01:09.684461+0000","last_fullsized":"2024-09-21T01:01:09.684461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685017+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:01:09.685017+0000","last_peered":"2024-09-21T01:01:09.685017+0000","last_clean":"2024-09-21T01:01:09.685017+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:01:09.685017+0000","last_undegraded":"2024-09-21T01:01:09.685017+0000","last_fullsized":"2024-09-21T01:01:09.685017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64692,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":161,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":185546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678044,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":515892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874273,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28680,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716472,"statfs":{"total":95995035648,"available":95965667328,"internally_reserved":0,"allocated":1695744,"data_stored":1163660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037800,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":520202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234029,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":516004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430259,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1617920,"data_stored":1112326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593785,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":511282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790015,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36928,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708224,"statfs":{"total":95995035648,"available":95957221376,"internally_reserved":0,"allocated":1622016,"data_stored":1103427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":516323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":511476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378704,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":568968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542231,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":567227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":512670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:02:02.883 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:02:03.299 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:03.752 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:03 smithi039 ceph-mon[30415]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:03.752 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:03 smithi039 ceph-mon[30415]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-09-21T01:02:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:03 smithi150 ceph-mon[34945]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:03 smithi150 ceph-mon[34945]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-09-21T01:02:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:03 smithi160 ceph-mon[34696]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:03 smithi160 ceph-mon[34696]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-09-21T01:02:03.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 -- 172.21.15.39:0/231551252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810b6f0 msgr2=0x7f8c5810bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 --2- 172.21.15.39:0/231551252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810b6f0 0x7f8c5810bb00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009900 tx=0x7f8c4802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 -- 172.21.15.39:0/231551252 shutdown_connections 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 --2- 172.21.15.39:0/231551252 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c5810c040 0x7f8c58112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 --2- 172.21.15.39:0/231551252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810b6f0 0x7f8c5810bb00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 --2- 172.21.15.39:0/231551252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c58105db0 0x7f8c581061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 -- 172.21.15.39:0/231551252 >> 172.21.15.39:0/231551252 conn(0x7f8c581017e0 msgr2=0x7f8c58103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 -- 172.21.15.39:0/231551252 shutdown_connections 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.931+0000 7f8c60800640 1 -- 172.21.15.39:0/231551252 wait complete. 2024-09-21T01:02:03.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.932+0000 7f8c60800640 1 Processor -- start 2024-09-21T01:02:03.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.932+0000 7f8c60800640 1 -- start start 2024-09-21T01:02:03.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c60800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c58105db0 0x7f8c5819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:03.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c60800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5810b6f0 0x7f8c5819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:03.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c60800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810c040 0x7f8c581a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:03.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c60800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8c58115d50 con 0x7f8c5810c040 2024-09-21T01:02:03.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c60800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8c58115bd0 con 0x7f8c5810b6f0 2024-09-21T01:02:03.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c60800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8c58115ed0 con 0x7f8c58105db0 2024-09-21T01:02:03.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c5e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810c040 0x7f8c581a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:03.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c5de00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c58105db0 0x7f8c5819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:03.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c5e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810c040 0x7f8c581a1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58844/0 (socket says 172.21.15.39:58844) 2024-09-21T01:02:03.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c5e800640 1 -- 172.21.15.39:0/3263523704 learned_addr learned my addr 172.21.15.39:0/3263523704 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:03.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c5d400640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5810b6f0 0x7f8c5819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:03.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.933+0000 7f8c5de00640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c58105db0 0x7f8c5819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37082/0 (socket says 172.21.15.39:37082) 2024-09-21T01:02:03.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c5d400640 1 -- 172.21.15.39:0/3263523704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c58105db0 msgr2=0x7f8c5819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c5d400640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c58105db0 0x7f8c5819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c5d400640 1 -- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810c040 msgr2=0x7f8c581a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c5d400640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810c040 0x7f8c581a1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c5d400640 1 -- 172.21.15.39:0/3263523704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c581a26f0 con 0x7f8c5810b6f0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c5d400640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5810b6f0 0x7f8c5819cd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009230 tx=0x7f8c48003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.934+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4803f070 con 0x7f8c5810b6f0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.935+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c581a2980 con 0x7f8c5810b6f0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.935+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c48031b50 con 0x7f8c5810b6f0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.935+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c581a9140 con 0x7f8c5810b6f0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.935+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48035030 con 0x7f8c5810b6f0 2024-09-21T01:02:03.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.936+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c581061c0 con 0x7f8c5810b6f0 2024-09-21T01:02:03.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.938+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f8c48042070 con 0x7f8c5810b6f0 2024-09-21T01:02:03.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.940+0000 7f8c52a00640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8c400b65f0 0x7f8c400b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:03.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.940+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f8c48108280 con 0x7f8c5810b6f0 2024-09-21T01:02:03.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.940+0000 7f8c5de00640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8c400b65f0 0x7f8c400b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:03.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.941+0000 7f8c5de00640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8c400b65f0 0x7f8c400b8b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009bd0 tx=0x7f8c4c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:03.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:03.942+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8c480d2900 con 0x7f8c5810b6f0 2024-09-21T01:02:04.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.137+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8c58063800 con 0x7f8c400b65f0 2024-09-21T01:02:04.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.144+0000 7f8c52a00640 1 -- 172.21.15.39:0/3263523704 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340353 (secure 0 0 0) 0x7f8c58063800 con 0x7f8c400b65f0 2024-09-21T01:02:04.150 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:04.152 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8c400b65f0 msgr2=0x7f8c400b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8c400b65f0 0x7f8c400b8b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009bd0 tx=0x7f8c4c005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5810b6f0 msgr2=0x7f8c5819cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5810b6f0 0x7f8c5819cd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009230 tx=0x7f8c48003b90 comp rx=0 tx=0).stop 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 shutdown_connections 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c5810c040 0x7f8c581a1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8c400b65f0 0x7f8c400b8b00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c5810b6f0 0x7f8c5819cd20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:04.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 --2- 172.21.15.39:0/3263523704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c58105db0 0x7f8c5819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:04.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.149+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 >> 172.21.15.39:0/3263523704 conn(0x7f8c581017e0 msgr2=0x7f8c5810ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:04.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.150+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 shutdown_connections 2024-09-21T01:02:04.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:04.150+0000 7f8c60800640 1 -- 172.21.15.39:0/3263523704 wait complete. 2024-09-21T01:02:05.262 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-21T01:02:03.907281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655294,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":228,"num_read_kb":206,"num_write":290,"num_write_kb":2265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371452,"kb_used_data":13884,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570372,"statfs":{"total":1151940427776,"available":1151560060928,"internally_reserved":0,"allocated":14217216,"data_stored":8123722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29140,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":24,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003611"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.351524+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:01:38.351524+0000","last_peered":"2024-09-21T01:01:38.351524+0000","last_clean":"2024-09-21T01:01:38.351524+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:01:38.351524+0000","last_undegraded":"2024-09-21T01:01:38.351524+0000","last_fullsized":"2024-09-21T01:01:38.351524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122696+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:01:10.122696+0000","last_peered":"2024-09-21T01:01:10.122696+0000","last_clean":"2024-09-21T01:01:10.122696+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:01:10.122696+0000","last_undegraded":"2024-09-21T01:01:10.122696+0000","last_fullsized":"2024-09-21T01:01:10.122696+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124288+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:01:10.124288+0000","last_peered":"2024-09-21T01:01:10.124288+0000","last_clean":"2024-09-21T01:01:10.124288+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:01:10.124288+0000","last_undegraded":"2024-09-21T01:01:10.124288+0000","last_fullsized":"2024-09-21T01:01:10.124288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.123049+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:01:10.123049+0000","last_peered":"2024-09-21T01:01:10.123049+0000","last_clean":"2024-09-21T01:01:10.123049+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:01:10.123049+0000","last_undegraded":"2024-09-21T01:01:10.123049+0000","last_fullsized":"2024-09-21T01:01:10.123049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684578+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:01:09.684578+0000","last_peered":"2024-09-21T01:01:09.684578+0000","last_clean":"2024-09-21T01:01:09.684578+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:01:09.684578+0000","last_undegraded":"2024-09-21T01:01:09.684578+0000","last_fullsized":"2024-09-21T01:01:09.684578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683982+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:01:09.683982+0000","last_peered":"2024-09-21T01:01:09.683982+0000","last_clean":"2024-09-21T01:01:09.683982+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:01:09.683982+0000","last_undegraded":"2024-09-21T01:01:09.683982+0000","last_fullsized":"2024-09-21T01:01:09.683982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.348489+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:01:38.348489+0000","last_peered":"2024-09-21T01:01:38.348489+0000","last_clean":"2024-09-21T01:01:38.348489+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:01:38.348489+0000","last_undegraded":"2024-09-21T01:01:38.348489+0000","last_fullsized":"2024-09-21T01:01:38.348489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685333+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:01:09.685333+0000","last_peered":"2024-09-21T01:01:09.685333+0000","last_clean":"2024-09-21T01:01:09.685333+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:01:09.685333+0000","last_undegraded":"2024-09-21T01:01:09.685333+0000","last_fullsized":"2024-09-21T01:01:09.685333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684503+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:01:09.684503+0000","last_peered":"2024-09-21T01:01:09.684503+0000","last_clean":"2024-09-21T01:01:09.684503+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:01:09.684503+0000","last_undegraded":"2024-09-21T01:01:09.684503+0000","last_fullsized":"2024-09-21T01:01:09.684503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681617+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:01:09.681617+0000","last_peered":"2024-09-21T01:01:09.681617+0000","last_clean":"2024-09-21T01:01:09.681617+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:01:09.681617+0000","last_undegraded":"2024-09-21T01:01:09.681617+0000","last_fullsized":"2024-09-21T01:01:09.681617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689792+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:01:09.689792+0000","last_peered":"2024-09-21T01:01:09.689792+0000","last_clean":"2024-09-21T01:01:09.689792+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:01:09.689792+0000","last_undegraded":"2024-09-21T01:01:09.689792+0000","last_fullsized":"2024-09-21T01:01:09.689792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684263+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:01:09.684263+0000","last_peered":"2024-09-21T01:01:09.684263+0000","last_clean":"2024-09-21T01:01:09.684263+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:01:09.684263+0000","last_undegraded":"2024-09-21T01:01:09.684263+0000","last_fullsized":"2024-09-21T01:01:09.684263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677837+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:01:09.677837+0000","last_peered":"2024-09-21T01:01:09.677837+0000","last_clean":"2024-09-21T01:01:09.677837+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:01:09.677837+0000","last_undegraded":"2024-09-21T01:01:09.677837+0000","last_fullsized":"2024-09-21T01:01:09.677837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.336046+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:01:37.336046+0000","last_peered":"2024-09-21T01:01:37.336046+0000","last_clean":"2024-09-21T01:01:37.336046+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:01:37.336046+0000","last_undegraded":"2024-09-21T01:01:37.336046+0000","last_fullsized":"2024-09-21T01:01:37.336046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684905+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:01:09.684905+0000","last_peered":"2024-09-21T01:01:09.684905+0000","last_clean":"2024-09-21T01:01:09.684905+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:01:09.684905+0000","last_undegraded":"2024-09-21T01:01:09.684905+0000","last_fullsized":"2024-09-21T01:01:09.684905+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122273+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:01:10.122273+0000","last_peered":"2024-09-21T01:01:10.122273+0000","last_clean":"2024-09-21T01:01:10.122273+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:01:10.122273+0000","last_undegraded":"2024-09-21T01:01:10.122273+0000","last_fullsized":"2024-09-21T01:01:10.122273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685448+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:01:09.685448+0000","last_peered":"2024-09-21T01:01:09.685448+0000","last_clean":"2024-09-21T01:01:09.685448+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:01:09.685448+0000","last_undegraded":"2024-09-21T01:01:09.685448+0000","last_fullsized":"2024-09-21T01:01:09.685448+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684346+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:01:09.684346+0000","last_peered":"2024-09-21T01:01:09.684346+0000","last_clean":"2024-09-21T01:01:09.684346+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:01:09.684346+0000","last_undegraded":"2024-09-21T01:01:09.684346+0000","last_fullsized":"2024-09-21T01:01:09.684346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683851+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:01:09.683851+0000","last_peered":"2024-09-21T01:01:09.683851+0000","last_clean":"2024-09-21T01:01:09.683851+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:01:09.683851+0000","last_undegraded":"2024-09-21T01:01:09.683851+0000","last_fullsized":"2024-09-21T01:01:09.683851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.340567+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:01:37.340567+0000","last_peered":"2024-09-21T01:01:37.340567+0000","last_clean":"2024-09-21T01:01:37.340567+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:01:37.340567+0000","last_undegraded":"2024-09-21T01:01:37.340567+0000","last_fullsized":"2024-09-21T01:01:37.340567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.123608+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:01:10.123608+0000","last_peered":"2024-09-21T01:01:10.123608+0000","last_clean":"2024-09-21T01:01:10.123608+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:01:10.123608+0000","last_undegraded":"2024-09-21T01:01:10.123608+0000","last_fullsized":"2024-09-21T01:01:10.123608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681124+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:01:09.681124+0000","last_peered":"2024-09-21T01:01:09.681124+0000","last_clean":"2024-09-21T01:01:09.681124+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:01:09.681124+0000","last_undegraded":"2024-09-21T01:01:09.681124+0000","last_fullsized":"2024-09-21T01:01:09.681124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.339021+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:01:37.339021+0000","last_peered":"2024-09-21T01:01:37.339021+0000","last_clean":"2024-09-21T01:01:37.339021+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:01:37.339021+0000","last_undegraded":"2024-09-21T01:01:37.339021+0000","last_fullsized":"2024-09-21T01:01:37.339021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684922+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:01:09.684922+0000","last_peered":"2024-09-21T01:01:09.684922+0000","last_clean":"2024-09-21T01:01:09.684922+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:01:09.684922+0000","last_undegraded":"2024-09-21T01:01:09.684922+0000","last_fullsized":"2024-09-21T01:01:09.684922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124259+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:01:10.124259+0000","last_peered":"2024-09-21T01:01:10.124259+0000","last_clean":"2024-09-21T01:01:10.124259+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:01:10.124259+0000","last_undegraded":"2024-09-21T01:01:10.124259+0000","last_fullsized":"2024-09-21T01:01:10.124259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.338547+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:01:37.338547+0000","last_peered":"2024-09-21T01:01:37.338547+0000","last_clean":"2024-09-21T01:01:37.338547+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:01:37.338547+0000","last_undegraded":"2024-09-21T01:01:37.338547+0000","last_fullsized":"2024-09-21T01:01:37.338547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685671+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:01:09.685671+0000","last_peered":"2024-09-21T01:01:09.685671+0000","last_clean":"2024-09-21T01:01:09.685671+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:01:09.685671+0000","last_undegraded":"2024-09-21T01:01:09.685671+0000","last_fullsized":"2024-09-21T01:01:09.685671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122813+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:01:10.122813+0000","last_peered":"2024-09-21T01:01:10.122813+0000","last_clean":"2024-09-21T01:01:10.122813+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:01:10.122813+0000","last_undegraded":"2024-09-21T01:01:10.122813+0000","last_fullsized":"2024-09-21T01:01:10.122813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:59.517166+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:01:59.517166+0000","last_peered":"2024-09-21T01:01:59.517166+0000","last_clean":"2024-09-21T01:01:59.517166+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:01:59.517166+0000","last_undegraded":"2024-09-21T01:01:59.517166+0000","last_fullsized":"2024-09-21T01:01:59.517166+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122152+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:01:10.122152+0000","last_peered":"2024-09-21T01:01:10.122152+0000","last_clean":"2024-09-21T01:01:10.122152+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:01:10.122152+0000","last_undegraded":"2024-09-21T01:01:10.122152+0000","last_fullsized":"2024-09-21T01:01:10.122152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681810+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:01:09.681810+0000","last_peered":"2024-09-21T01:01:09.681810+0000","last_clean":"2024-09-21T01:01:09.681810+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:01:09.681810+0000","last_undegraded":"2024-09-21T01:01:09.681810+0000","last_fullsized":"2024-09-21T01:01:09.681810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.338712+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:01:37.338712+0000","last_peered":"2024-09-21T01:01:37.338712+0000","last_clean":"2024-09-21T01:01:37.338712+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:01:37.338712+0000","last_undegraded":"2024-09-21T01:01:37.338712+0000","last_fullsized":"2024-09-21T01:01:37.338712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683672+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:01:09.683672+0000","last_peered":"2024-09-21T01:01:09.683672+0000","last_clean":"2024-09-21T01:01:09.683672+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:01:09.683672+0000","last_undegraded":"2024-09-21T01:01:09.683672+0000","last_fullsized":"2024-09-21T01:01:09.683672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925507+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:01:09.925507+0000","last_peered":"2024-09-21T01:01:09.925507+0000","last_clean":"2024-09-21T01:01:09.925507+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:01:09.925507+0000","last_undegraded":"2024-09-21T01:01:09.925507+0000","last_fullsized":"2024-09-21T01:01:09.925507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124130+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:01:10.124130+0000","last_peered":"2024-09-21T01:01:10.124130+0000","last_clean":"2024-09-21T01:01:10.124130+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:01:10.124130+0000","last_undegraded":"2024-09-21T01:01:10.124130+0000","last_fullsized":"2024-09-21T01:01:10.124130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681155+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:01:09.681155+0000","last_peered":"2024-09-21T01:01:09.681155+0000","last_clean":"2024-09-21T01:01:09.681155+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:01:09.681155+0000","last_undegraded":"2024-09-21T01:01:09.681155+0000","last_fullsized":"2024-09-21T01:01:09.681155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925284+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:01:09.925284+0000","last_peered":"2024-09-21T01:01:09.925284+0000","last_clean":"2024-09-21T01:01:09.925284+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:01:09.925284+0000","last_undegraded":"2024-09-21T01:01:09.925284+0000","last_fullsized":"2024-09-21T01:01:09.925284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678032+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:01:09.678032+0000","last_peered":"2024-09-21T01:01:09.678032+0000","last_clean":"2024-09-21T01:01:09.678032+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:01:09.678032+0000","last_undegraded":"2024-09-21T01:01:09.678032+0000","last_fullsized":"2024-09-21T01:01:09.678032+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681699+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:01:09.681699+0000","last_peered":"2024-09-21T01:01:09.681699+0000","last_clean":"2024-09-21T01:01:09.681699+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:01:09.681699+0000","last_undegraded":"2024-09-21T01:01:09.681699+0000","last_fullsized":"2024-09-21T01:01:09.681699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.691419+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:01:09.691419+0000","last_peered":"2024-09-21T01:01:09.691419+0000","last_clean":"2024-09-21T01:01:09.691419+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:01:09.691419+0000","last_undegraded":"2024-09-21T01:01:09.691419+0000","last_fullsized":"2024-09-21T01:01:09.691419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:02:00.410652+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:02:00.410652+0000","last_peered":"2024-09-21T01:02:00.410652+0000","last_clean":"2024-09-21T01:02:00.410652+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:02:00.410652+0000","last_undegraded":"2024-09-21T01:02:00.410652+0000","last_fullsized":"2024-09-21T01:02:00.410652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681687+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:01:09.681687+0000","last_peered":"2024-09-21T01:01:09.681687+0000","last_clean":"2024-09-21T01:01:09.681687+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:01:09.681687+0000","last_undegraded":"2024-09-21T01:01:09.681687+0000","last_fullsized":"2024-09-21T01:01:09.681687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681079+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:01:09.681079+0000","last_peered":"2024-09-21T01:01:09.681079+0000","last_clean":"2024-09-21T01:01:09.681079+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:01:09.681079+0000","last_undegraded":"2024-09-21T01:01:09.681079+0000","last_fullsized":"2024-09-21T01:01:09.681079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.348209+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:01:38.348209+0000","last_peered":"2024-09-21T01:01:38.348209+0000","last_clean":"2024-09-21T01:01:38.348209+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:01:38.348209+0000","last_undegraded":"2024-09-21T01:01:38.348209+0000","last_fullsized":"2024-09-21T01:01:38.348209+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.688633+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:01:09.688633+0000","last_peered":"2024-09-21T01:01:09.688633+0000","last_clean":"2024-09-21T01:01:09.688633+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:01:09.688633+0000","last_undegraded":"2024-09-21T01:01:09.688633+0000","last_fullsized":"2024-09-21T01:01:09.688633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122941+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:01:10.122941+0000","last_peered":"2024-09-21T01:01:10.122941+0000","last_clean":"2024-09-21T01:01:10.122941+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:01:10.122941+0000","last_undegraded":"2024-09-21T01:01:10.122941+0000","last_fullsized":"2024-09-21T01:01:10.122941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:41.370435+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:01:41.370435+0000","last_peered":"2024-09-21T01:01:41.370435+0000","last_clean":"2024-09-21T01:01:41.370435+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:01:41.370435+0000","last_undegraded":"2024-09-21T01:01:41.370435+0000","last_fullsized":"2024-09-21T01:01:41.370435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678560+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:01:09.678560+0000","last_peered":"2024-09-21T01:01:09.678560+0000","last_clean":"2024-09-21T01:01:09.678560+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:01:09.678560+0000","last_undegraded":"2024-09-21T01:01:09.678560+0000","last_fullsized":"2024-09-21T01:01:09.678560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.125560+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:01:10.125560+0000","last_peered":"2024-09-21T01:01:10.125560+0000","last_clean":"2024-09-21T01:01:10.125560+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:01:10.125560+0000","last_undegraded":"2024-09-21T01:01:10.125560+0000","last_fullsized":"2024-09-21T01:01:10.125560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678540+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:01:09.678540+0000","last_peered":"2024-09-21T01:01:09.678540+0000","last_clean":"2024-09-21T01:01:09.678540+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:01:09.678540+0000","last_undegraded":"2024-09-21T01:01:09.678540+0000","last_fullsized":"2024-09-21T01:01:09.678540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.351056+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:01:38.351056+0000","last_peered":"2024-09-21T01:01:38.351056+0000","last_clean":"2024-09-21T01:01:38.351056+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:01:38.351056+0000","last_undegraded":"2024-09-21T01:01:38.351056+0000","last_fullsized":"2024-09-21T01:01:38.351056+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684715+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:01:09.684715+0000","last_peered":"2024-09-21T01:01:09.684715+0000","last_clean":"2024-09-21T01:01:09.684715+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:01:09.684715+0000","last_undegraded":"2024-09-21T01:01:09.684715+0000","last_fullsized":"2024-09-21T01:01:09.684715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.126961+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:01:10.126961+0000","last_peered":"2024-09-21T01:01:10.126961+0000","last_clean":"2024-09-21T01:01:10.126961+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:01:10.126961+0000","last_undegraded":"2024-09-21T01:01:10.126961+0000","last_fullsized":"2024-09-21T01:01:10.126961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.690046+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:01:09.690046+0000","last_peered":"2024-09-21T01:01:09.690046+0000","last_clean":"2024-09-21T01:01:09.690046+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:01:09.690046+0000","last_undegraded":"2024-09-21T01:01:09.690046+0000","last_fullsized":"2024-09-21T01:01:09.690046+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684349+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:01:09.684349+0000","last_peered":"2024-09-21T01:01:09.684349+0000","last_clean":"2024-09-21T01:01:09.684349+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:01:09.684349+0000","last_undegraded":"2024-09-21T01:01:09.684349+0000","last_fullsized":"2024-09-21T01:01:09.684349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678057+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:01:09.678057+0000","last_peered":"2024-09-21T01:01:09.678057+0000","last_clean":"2024-09-21T01:01:09.678057+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:01:09.678057+0000","last_undegraded":"2024-09-21T01:01:09.678057+0000","last_fullsized":"2024-09-21T01:01:09.678057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681594+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:01:09.681594+0000","last_peered":"2024-09-21T01:01:09.681594+0000","last_clean":"2024-09-21T01:01:09.681594+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:01:09.681594+0000","last_undegraded":"2024-09-21T01:01:09.681594+0000","last_fullsized":"2024-09-21T01:01:09.681594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689297+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:01:09.689297+0000","last_peered":"2024-09-21T01:01:09.689297+0000","last_clean":"2024-09-21T01:01:09.689297+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:01:09.689297+0000","last_undegraded":"2024-09-21T01:01:09.689297+0000","last_fullsized":"2024-09-21T01:01:09.689297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684237+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:01:09.684237+0000","last_peered":"2024-09-21T01:01:09.684237+0000","last_clean":"2024-09-21T01:01:09.684237+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:01:09.684237+0000","last_undegraded":"2024-09-21T01:01:09.684237+0000","last_fullsized":"2024-09-21T01:01:09.684237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:40.345501+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:01:40.345501+0000","last_peered":"2024-09-21T01:01:40.345501+0000","last_clean":"2024-09-21T01:01:40.345501+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:01:40.345501+0000","last_undegraded":"2024-09-21T01:01:40.345501+0000","last_fullsized":"2024-09-21T01:01:40.345501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.687731+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:01:09.687731+0000","last_peered":"2024-09-21T01:01:09.687731+0000","last_clean":"2024-09-21T01:01:09.687731+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:01:09.687731+0000","last_undegraded":"2024-09-21T01:01:09.687731+0000","last_fullsized":"2024-09-21T01:01:09.687731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.687911+0000","last_change":"2024-09-21T01:00:42.495651+0000","last_active":"2024-09-21T01:01:09.687911+0000","last_peered":"2024-09-21T01:01:09.687911+0000","last_clean":"2024-09-21T01:01:09.687911+0000","last_became_active":"2024-09-21T01:00:42.495096+0000","last_became_peered":"2024-09-21T01:00:42.495096+0000","last_unstale":"2024-09-21T01:01:09.687911+0000","last_undegraded":"2024-09-21T01:01:09.687911+0000","last_fullsized":"2024-09-21T01:01:09.687911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:19.981242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.354362+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:01:38.354362+0000","last_peered":"2024-09-21T01:01:38.354362+0000","last_clean":"2024-09-21T01:01:38.354362+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:01:38.354362+0000","last_undegraded":"2024-09-21T01:01:38.354362+0000","last_fullsized":"2024-09-21T01:01:38.354362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681482+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:01:09.681482+0000","last_peered":"2024-09-21T01:01:09.681482+0000","last_clean":"2024-09-21T01:01:09.681482+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:01:09.681482+0000","last_undegraded":"2024-09-21T01:01:09.681482+0000","last_fullsized":"2024-09-21T01:01:09.681482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677752+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:01:09.677752+0000","last_peered":"2024-09-21T01:01:09.677752+0000","last_clean":"2024-09-21T01:01:09.677752+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:01:09.677752+0000","last_undegraded":"2024-09-21T01:01:09.677752+0000","last_fullsized":"2024-09-21T01:01:09.677752+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:02:00.411281+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:02:00.411281+0000","last_peered":"2024-09-21T01:02:00.411281+0000","last_clean":"2024-09-21T01:02:00.411281+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:02:00.411281+0000","last_undegraded":"2024-09-21T01:02:00.411281+0000","last_fullsized":"2024-09-21T01:02:00.411281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.125687+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:01:10.125687+0000","last_peered":"2024-09-21T01:01:10.125687+0000","last_clean":"2024-09-21T01:01:10.125687+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:01:10.125687+0000","last_undegraded":"2024-09-21T01:01:10.125687+0000","last_fullsized":"2024-09-21T01:01:10.125687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677991+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:01:09.677991+0000","last_peered":"2024-09-21T01:01:09.677991+0000","last_clean":"2024-09-21T01:01:09.677991+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:01:09.677991+0000","last_undegraded":"2024-09-21T01:01:09.677991+0000","last_fullsized":"2024-09-21T01:01:09.677991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.339054+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:01:37.339054+0000","last_peered":"2024-09-21T01:01:37.339054+0000","last_clean":"2024-09-21T01:01:37.339054+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:01:37.339054+0000","last_undegraded":"2024-09-21T01:01:37.339054+0000","last_fullsized":"2024-09-21T01:01:37.339054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683501+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:01:09.683501+0000","last_peered":"2024-09-21T01:01:09.683501+0000","last_clean":"2024-09-21T01:01:09.683501+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:01:09.683501+0000","last_undegraded":"2024-09-21T01:01:09.683501+0000","last_fullsized":"2024-09-21T01:01:09.683501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681297+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:01:09.681297+0000","last_peered":"2024-09-21T01:01:09.681297+0000","last_clean":"2024-09-21T01:01:09.681297+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:01:09.681297+0000","last_undegraded":"2024-09-21T01:01:09.681297+0000","last_fullsized":"2024-09-21T01:01:09.681297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.353414+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:01:38.353414+0000","last_peered":"2024-09-21T01:01:38.353414+0000","last_clean":"2024-09-21T01:01:38.353414+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:01:38.353414+0000","last_undegraded":"2024-09-21T01:01:38.353414+0000","last_fullsized":"2024-09-21T01:01:38.353414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689729+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:01:09.689729+0000","last_peered":"2024-09-21T01:01:09.689729+0000","last_clean":"2024-09-21T01:01:09.689729+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:01:09.689729+0000","last_undegraded":"2024-09-21T01:01:09.689729+0000","last_fullsized":"2024-09-21T01:01:09.689729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681585+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:01:09.681585+0000","last_peered":"2024-09-21T01:01:09.681585+0000","last_clean":"2024-09-21T01:01:09.681585+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:01:09.681585+0000","last_undegraded":"2024-09-21T01:01:09.681585+0000","last_fullsized":"2024-09-21T01:01:09.681585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'30","reported_seq":115,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:59.410258+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:01:59.410258+0000","last_peered":"2024-09-21T01:01:59.410258+0000","last_clean":"2024-09-21T01:01:59.410258+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:01:59.410258+0000","last_undegraded":"2024-09-21T01:01:59.410258+0000","last_fullsized":"2024-09-21T01:01:59.410258+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60300,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":77,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124323+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:01:10.124323+0000","last_peered":"2024-09-21T01:01:10.124323+0000","last_clean":"2024-09-21T01:01:10.124323+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:01:10.124323+0000","last_undegraded":"2024-09-21T01:01:10.124323+0000","last_fullsized":"2024-09-21T01:01:10.124323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685501+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:01:09.685501+0000","last_peered":"2024-09-21T01:01:09.685501+0000","last_clean":"2024-09-21T01:01:09.685501+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:01:09.685501+0000","last_undegraded":"2024-09-21T01:01:09.685501+0000","last_fullsized":"2024-09-21T01:01:09.685501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.338691+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:01:37.338691+0000","last_peered":"2024-09-21T01:01:37.338691+0000","last_clean":"2024-09-21T01:01:37.338691+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:01:37.338691+0000","last_undegraded":"2024-09-21T01:01:37.338691+0000","last_fullsized":"2024-09-21T01:01:37.338691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689108+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:01:09.689108+0000","last_peered":"2024-09-21T01:01:09.689108+0000","last_clean":"2024-09-21T01:01:09.689108+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:01:09.689108+0000","last_undegraded":"2024-09-21T01:01:09.689108+0000","last_fullsized":"2024-09-21T01:01:09.689108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684603+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:01:09.684603+0000","last_peered":"2024-09-21T01:01:09.684603+0000","last_clean":"2024-09-21T01:01:09.684603+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:01:09.684603+0000","last_undegraded":"2024-09-21T01:01:09.684603+0000","last_fullsized":"2024-09-21T01:01:09.684603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:37.337512+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:01:37.337512+0000","last_peered":"2024-09-21T01:01:37.337512+0000","last_clean":"2024-09-21T01:01:37.337512+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:01:37.337512+0000","last_undegraded":"2024-09-21T01:01:37.337512+0000","last_fullsized":"2024-09-21T01:01:37.337512+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.121919+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:01:10.121919+0000","last_peered":"2024-09-21T01:01:10.121919+0000","last_clean":"2024-09-21T01:01:10.121919+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:01:10.121919+0000","last_undegraded":"2024-09-21T01:01:10.121919+0000","last_fullsized":"2024-09-21T01:01:10.121919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685964+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:01:09.685964+0000","last_peered":"2024-09-21T01:01:09.685964+0000","last_clean":"2024-09-21T01:01:09.685964+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:01:09.685964+0000","last_undegraded":"2024-09-21T01:01:09.685964+0000","last_fullsized":"2024-09-21T01:01:09.685964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:58.402318+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:01:58.402318+0000","last_peered":"2024-09-21T01:01:58.402318+0000","last_clean":"2024-09-21T01:01:58.402318+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:01:58.402318+0000","last_undegraded":"2024-09-21T01:01:58.402318+0000","last_fullsized":"2024-09-21T01:01:58.402318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122398+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:01:10.122398+0000","last_peered":"2024-09-21T01:01:10.122398+0000","last_clean":"2024-09-21T01:01:10.122398+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:01:10.122398+0000","last_undegraded":"2024-09-21T01:01:10.122398+0000","last_fullsized":"2024-09-21T01:01:10.122398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684337+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:01:09.684337+0000","last_peered":"2024-09-21T01:01:09.684337+0000","last_clean":"2024-09-21T01:01:09.684337+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:01:09.684337+0000","last_undegraded":"2024-09-21T01:01:09.684337+0000","last_fullsized":"2024-09-21T01:01:09.684337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.123309+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:01:10.123309+0000","last_peered":"2024-09-21T01:01:10.123309+0000","last_clean":"2024-09-21T01:01:10.123309+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:01:10.123309+0000","last_undegraded":"2024-09-21T01:01:10.123309+0000","last_fullsized":"2024-09-21T01:01:10.123309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.689076+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:01:09.689076+0000","last_peered":"2024-09-21T01:01:09.689076+0000","last_clean":"2024-09-21T01:01:09.689076+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:01:09.689076+0000","last_undegraded":"2024-09-21T01:01:09.689076+0000","last_fullsized":"2024-09-21T01:01:09.689076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.691298+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:01:09.691298+0000","last_peered":"2024-09-21T01:01:09.691298+0000","last_clean":"2024-09-21T01:01:09.691298+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:01:09.691298+0000","last_undegraded":"2024-09-21T01:01:09.691298+0000","last_fullsized":"2024-09-21T01:01:09.691298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122088+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:01:10.122088+0000","last_peered":"2024-09-21T01:01:10.122088+0000","last_clean":"2024-09-21T01:01:10.122088+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:01:10.122088+0000","last_undegraded":"2024-09-21T01:01:10.122088+0000","last_fullsized":"2024-09-21T01:01:10.122088+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.352664+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:01:38.352664+0000","last_peered":"2024-09-21T01:01:38.352664+0000","last_clean":"2024-09-21T01:01:38.352664+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:01:38.352664+0000","last_undegraded":"2024-09-21T01:01:38.352664+0000","last_fullsized":"2024-09-21T01:01:38.352664+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:38.351883+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:01:38.351883+0000","last_peered":"2024-09-21T01:01:38.351883+0000","last_clean":"2024-09-21T01:01:38.351883+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:01:38.351883+0000","last_undegraded":"2024-09-21T01:01:38.351883+0000","last_fullsized":"2024-09-21T01:01:38.351883+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.688239+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:01:09.688239+0000","last_peered":"2024-09-21T01:01:09.688239+0000","last_clean":"2024-09-21T01:01:09.688239+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:01:09.688239+0000","last_undegraded":"2024-09-21T01:01:09.688239+0000","last_fullsized":"2024-09-21T01:01:09.688239+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685565+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:01:09.685565+0000","last_peered":"2024-09-21T01:01:09.685565+0000","last_clean":"2024-09-21T01:01:09.685565+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:01:09.685565+0000","last_undegraded":"2024-09-21T01:01:09.685565+0000","last_fullsized":"2024-09-21T01:01:09.685565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:02:00.406497+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:02:00.406497+0000","last_peered":"2024-09-21T01:02:00.406497+0000","last_clean":"2024-09-21T01:02:00.406497+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:02:00.406497+0000","last_undegraded":"2024-09-21T01:02:00.406497+0000","last_fullsized":"2024-09-21T01:02:00.406497+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681227+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:01:09.681227+0000","last_peered":"2024-09-21T01:01:09.681227+0000","last_clean":"2024-09-21T01:01:09.681227+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:01:09.681227+0000","last_undegraded":"2024-09-21T01:01:09.681227+0000","last_fullsized":"2024-09-21T01:01:09.681227+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.686184+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:01:09.686184+0000","last_peered":"2024-09-21T01:01:09.686184+0000","last_clean":"2024-09-21T01:01:09.686184+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:01:09.686184+0000","last_undegraded":"2024-09-21T01:01:09.686184+0000","last_fullsized":"2024-09-21T01:01:09.686184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683461+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:01:09.683461+0000","last_peered":"2024-09-21T01:01:09.683461+0000","last_clean":"2024-09-21T01:01:09.683461+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:01:09.683461+0000","last_undegraded":"2024-09-21T01:01:09.683461+0000","last_fullsized":"2024-09-21T01:01:09.683461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685864+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:01:09.685864+0000","last_peered":"2024-09-21T01:01:09.685864+0000","last_clean":"2024-09-21T01:01:09.685864+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:01:09.685864+0000","last_undegraded":"2024-09-21T01:01:09.685864+0000","last_fullsized":"2024-09-21T01:01:09.685864+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685275+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:01:09.685275+0000","last_peered":"2024-09-21T01:01:09.685275+0000","last_clean":"2024-09-21T01:01:09.685275+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:01:09.685275+0000","last_undegraded":"2024-09-21T01:01:09.685275+0000","last_fullsized":"2024-09-21T01:01:09.685275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684461+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:01:09.684461+0000","last_peered":"2024-09-21T01:01:09.684461+0000","last_clean":"2024-09-21T01:01:09.684461+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:01:09.684461+0000","last_undegraded":"2024-09-21T01:01:09.684461+0000","last_fullsized":"2024-09-21T01:01:09.684461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681765+0000","last_change":"2024-09-21T01:00:42.500918+0000","last_active":"2024-09-21T01:01:09.681765+0000","last_peered":"2024-09-21T01:01:09.681765+0000","last_clean":"2024-09-21T01:01:09.681765+0000","last_became_active":"2024-09-21T01:00:42.497579+0000","last_became_peered":"2024-09-21T01:00:42.497579+0000","last_unstale":"2024-09-21T01:01:09.681765+0000","last_undegraded":"2024-09-21T01:01:09.681765+0000","last_fullsized":"2024-09-21T01:01:09.681765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:43:22.447476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684199+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:01:09.684199+0000","last_peered":"2024-09-21T01:01:09.684199+0000","last_clean":"2024-09-21T01:01:09.684199+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:01:09.684199+0000","last_undegraded":"2024-09-21T01:01:09.684199+0000","last_fullsized":"2024-09-21T01:01:09.684199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685747+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:01:09.685747+0000","last_peered":"2024-09-21T01:01:09.685747+0000","last_clean":"2024-09-21T01:01:09.685747+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:01:09.685747+0000","last_undegraded":"2024-09-21T01:01:09.685747+0000","last_fullsized":"2024-09-21T01:01:09.685747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122642+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:01:10.122642+0000","last_peered":"2024-09-21T01:01:10.122642+0000","last_clean":"2024-09-21T01:01:10.122642+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:01:10.122642+0000","last_undegraded":"2024-09-21T01:01:10.122642+0000","last_fullsized":"2024-09-21T01:01:10.122642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.124744+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:01:10.124744+0000","last_peered":"2024-09-21T01:01:10.124744+0000","last_clean":"2024-09-21T01:01:10.124744+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:01:10.124744+0000","last_undegraded":"2024-09-21T01:01:10.124744+0000","last_fullsized":"2024-09-21T01:01:10.124744+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.122734+0000","last_change":"2024-09-21T01:00:42.493698+0000","last_active":"2024-09-21T01:01:10.122734+0000","last_peered":"2024-09-21T01:01:10.122734+0000","last_clean":"2024-09-21T01:01:10.122734+0000","last_became_active":"2024-09-21T01:00:42.493338+0000","last_became_peered":"2024-09-21T01:00:42.493338+0000","last_unstale":"2024-09-21T01:01:10.122734+0000","last_undegraded":"2024-09-21T01:01:10.122734+0000","last_fullsized":"2024-09-21T01:01:10.122734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:03.505872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685357+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:01:09.685357+0000","last_peered":"2024-09-21T01:01:09.685357+0000","last_clean":"2024-09-21T01:01:09.685357+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:01:09.685357+0000","last_undegraded":"2024-09-21T01:01:09.685357+0000","last_fullsized":"2024-09-21T01:01:09.685357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681522+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:01:09.681522+0000","last_peered":"2024-09-21T01:01:09.681522+0000","last_clean":"2024-09-21T01:01:09.681522+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:01:09.681522+0000","last_undegraded":"2024-09-21T01:01:09.681522+0000","last_fullsized":"2024-09-21T01:01:09.681522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.683546+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:01:09.683546+0000","last_peered":"2024-09-21T01:01:09.683546+0000","last_clean":"2024-09-21T01:01:09.683546+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:01:09.683546+0000","last_undegraded":"2024-09-21T01:01:09.683546+0000","last_fullsized":"2024-09-21T01:01:09.683546+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.686367+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:01:09.686367+0000","last_peered":"2024-09-21T01:01:09.686367+0000","last_clean":"2024-09-21T01:01:09.686367+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:01:09.686367+0000","last_undegraded":"2024-09-21T01:01:09.686367+0000","last_fullsized":"2024-09-21T01:01:09.686367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.125200+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:01:10.125200+0000","last_peered":"2024-09-21T01:01:10.125200+0000","last_clean":"2024-09-21T01:01:10.125200+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:01:10.125200+0000","last_undegraded":"2024-09-21T01:01:10.125200+0000","last_fullsized":"2024-09-21T01:01:10.125200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685144+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:01:09.685144+0000","last_peered":"2024-09-21T01:01:09.685144+0000","last_clean":"2024-09-21T01:01:09.685144+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:01:09.685144+0000","last_undegraded":"2024-09-21T01:01:09.685144+0000","last_fullsized":"2024-09-21T01:01:09.685144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684137+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:01:09.684137+0000","last_peered":"2024-09-21T01:01:09.684137+0000","last_clean":"2024-09-21T01:01:09.684137+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:01:09.684137+0000","last_undegraded":"2024-09-21T01:01:09.684137+0000","last_fullsized":"2024-09-21T01:01:09.684137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684072+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:01:09.684072+0000","last_peered":"2024-09-21T01:01:09.684072+0000","last_clean":"2024-09-21T01:01:09.684072+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:01:09.684072+0000","last_undegraded":"2024-09-21T01:01:09.684072+0000","last_fullsized":"2024-09-21T01:01:09.684072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684180+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:01:09.684180+0000","last_peered":"2024-09-21T01:01:09.684180+0000","last_clean":"2024-09-21T01:01:09.684180+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:01:09.684180+0000","last_undegraded":"2024-09-21T01:01:09.684180+0000","last_fullsized":"2024-09-21T01:01:09.684180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681482+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:01:09.681482+0000","last_peered":"2024-09-21T01:01:09.681482+0000","last_clean":"2024-09-21T01:01:09.681482+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:01:09.681482+0000","last_undegraded":"2024-09-21T01:01:09.681482+0000","last_fullsized":"2024-09-21T01:01:09.681482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684764+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:01:09.684764+0000","last_peered":"2024-09-21T01:01:09.684764+0000","last_clean":"2024-09-21T01:01:09.684764+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:01:09.684764+0000","last_undegraded":"2024-09-21T01:01:09.684764+0000","last_fullsized":"2024-09-21T01:01:09.684764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.678366+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:01:09.678366+0000","last_peered":"2024-09-21T01:01:09.678366+0000","last_clean":"2024-09-21T01:01:09.678366+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:01:09.678366+0000","last_undegraded":"2024-09-21T01:01:09.678366+0000","last_fullsized":"2024-09-21T01:01:09.678366+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925537+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:01:09.925537+0000","last_peered":"2024-09-21T01:01:09.925537+0000","last_clean":"2024-09-21T01:01:09.925537+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:01:09.925537+0000","last_undegraded":"2024-09-21T01:01:09.925537+0000","last_fullsized":"2024-09-21T01:01:09.925537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685903+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:01:09.685903+0000","last_peered":"2024-09-21T01:01:09.685903+0000","last_clean":"2024-09-21T01:01:09.685903+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:01:09.685903+0000","last_undegraded":"2024-09-21T01:01:09.685903+0000","last_fullsized":"2024-09-21T01:01:09.685903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:10.121968+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:01:10.121968+0000","last_peered":"2024-09-21T01:01:10.121968+0000","last_clean":"2024-09-21T01:01:10.121968+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:01:10.121968+0000","last_undegraded":"2024-09-21T01:01:10.121968+0000","last_fullsized":"2024-09-21T01:01:10.121968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.677767+0000","last_change":"2024-09-21T01:00:42.498438+0000","last_active":"2024-09-21T01:01:09.677767+0000","last_peered":"2024-09-21T01:01:09.677767+0000","last_clean":"2024-09-21T01:01:09.677767+0000","last_became_active":"2024-09-21T01:00:42.498080+0000","last_became_peered":"2024-09-21T01:00:42.498080+0000","last_unstale":"2024-09-21T01:01:09.677767+0000","last_undegraded":"2024-09-21T01:01:09.677767+0000","last_fullsized":"2024-09-21T01:01:09.677767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:23.853505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.925111+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:01:09.925111+0000","last_peered":"2024-09-21T01:01:09.925111+0000","last_clean":"2024-09-21T01:01:09.925111+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:01:09.925111+0000","last_undegraded":"2024-09-21T01:01:09.925111+0000","last_fullsized":"2024-09-21T01:01:09.925111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.681265+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:01:09.681265+0000","last_peered":"2024-09-21T01:01:09.681265+0000","last_clean":"2024-09-21T01:01:09.681265+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:01:09.681265+0000","last_undegraded":"2024-09-21T01:01:09.681265+0000","last_fullsized":"2024-09-21T01:01:09.681265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684599+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:01:09.684599+0000","last_peered":"2024-09-21T01:01:09.684599+0000","last_clean":"2024-09-21T01:01:09.684599+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:01:09.684599+0000","last_undegraded":"2024-09-21T01:01:09.684599+0000","last_fullsized":"2024-09-21T01:01:09.684599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684305+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:01:09.684305+0000","last_peered":"2024-09-21T01:01:09.684305+0000","last_clean":"2024-09-21T01:01:09.684305+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:01:09.684305+0000","last_undegraded":"2024-09-21T01:01:09.684305+0000","last_fullsized":"2024-09-21T01:01:09.684305+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.684461+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:01:09.684461+0000","last_peered":"2024-09-21T01:01:09.684461+0000","last_clean":"2024-09-21T01:01:09.684461+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:01:09.684461+0000","last_undegraded":"2024-09-21T01:01:09.684461+0000","last_fullsized":"2024-09-21T01:01:09.684461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-21T01:01:09.685017+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:01:09.685017+0000","last_peered":"2024-09-21T01:01:09.685017+0000","last_clean":"2024-09-21T01:01:09.685017+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:01:09.685017+0000","last_undegraded":"2024-09-21T01:01:09.685017+0000","last_fullsized":"2024-09-21T01:01:09.685017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64692,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":147,"num_read_kb":137,"num_write":165,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":189811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678045,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":515892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874273,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28680,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716472,"statfs":{"total":95995035648,"available":95965667328,"internally_reserved":0,"allocated":1695744,"data_stored":1163660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037801,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":520202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234030,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":516004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430259,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1617920,"data_stored":1112326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593786,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":511282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36928,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708224,"statfs":{"total":95995035648,"available":95957221376,"internally_reserved":0,"allocated":1622016,"data_stored":1103427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953542,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":516323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":511476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378705,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":573233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542231,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":567227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":512670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:02:05.267 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T01:02:05.267 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T01:02:05.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T01:02:05.649 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:05 smithi039 ceph-mon[30415]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-09-21T01:02:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:05 smithi039 ceph-mon[30415]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:05 smithi150 ceph-mon[34945]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-09-21T01:02:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:05 smithi150 ceph-mon[34945]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:05 smithi160 ceph-mon[34696]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-09-21T01:02:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:05 smithi160 ceph-mon[34696]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:06.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 -- 172.21.15.39:0/3907411801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 msgr2=0x7f6454103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:06.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 --2- 172.21.15.39:0/3907411801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f6454103f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6444009900 tx=0x7f644402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 -- 172.21.15.39:0/3907411801 shutdown_connections 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 --2- 172.21.15.39:0/3907411801 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6454109e00 0x7f6454110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 --2- 172.21.15.39:0/3907411801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541094b0 0x7f64541098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 --2- 172.21.15.39:0/3907411801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f6454103f80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 -- 172.21.15.39:0/3907411801 >> 172.21.15.39:0/3907411801 conn(0x7f64540ff5a0 msgr2=0x7f64541019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 -- 172.21.15.39:0/3907411801 shutdown_connections 2024-09-21T01:02:06.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.274+0000 7f6459a00640 1 -- 172.21.15.39:0/3907411801 wait complete. 2024-09-21T01:02:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.275+0000 7f6459a00640 1 Processor -- start 2024-09-21T01:02:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.275+0000 7f6459a00640 1 -- start start 2024-09-21T01:02:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.275+0000 7f6459a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f645419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6459a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64541094b0 0x7f645419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6459a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6454109e00 0x7f64541a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6459a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6454113ac0 con 0x7f6454103b70 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6459a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6454113940 con 0x7f6454109e00 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6459a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6454113c40 con 0x7f64541094b0 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f645419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64541094b0 0x7f645419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f645419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58858/0 (socket says 172.21.15.39:58858) 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 -- 172.21.15.39:0/1762780780 learned_addr learned my addr 172.21.15.39:0/1762780780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:06.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6453400640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6454109e00 0x7f64541a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 -- 172.21.15.39:0/1762780780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64541094b0 msgr2=0x7f645419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64541094b0 0x7f645419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 -- 172.21.15.39:0/1762780780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6454109e00 msgr2=0x7f64541a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6454109e00 0x7f64541a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.276+0000 7f6452a00640 1 -- 172.21.15.39:0/1762780780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64541a2640 con 0x7f6454103b70 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.277+0000 7f6452a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f645419c6d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6444005940 tx=0x7f64440040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.277+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644403f030 con 0x7f6454103b70 2024-09-21T01:02:06.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.277+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64440044c0 con 0x7f6454103b70 2024-09-21T01:02:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.277+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64541a28d0 con 0x7f6454103b70 2024-09-21T01:02:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.277+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444035070 con 0x7f6454103b70 2024-09-21T01:02:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.277+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64541a9090 con 0x7f6454103b70 2024-09-21T01:02:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.279+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6454103f80 con 0x7f6454103b70 2024-09-21T01:02:06.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.281+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f6444042030 con 0x7f6454103b70 2024-09-21T01:02:06.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.282+0000 7f644b400640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64380b65f0 0x7f64380b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.282+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f6444107cc0 con 0x7f6454103b70 2024-09-21T01:02:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.283+0000 7f6452000640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64380b65f0 0x7f64380b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.283+0000 7f6452000640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64380b65f0 0x7f64380b8b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f645406b2a0 tx=0x7f6440005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:06.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.287+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f644410e080 con 0x7f6454103b70 2024-09-21T01:02:06.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.560+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f64541a9290 con 0x7f6454103b70 2024-09-21T01:02:06.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.567+0000 7f644b400640 1 -- 172.21.15.39:0/1762780780 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f64440d2340 con 0x7f6454103b70 2024-09-21T01:02:06.568 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-09-21T01:02:06.569 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAOG+5maIGDIRAAqVhJ5p9qTpaOBScembnUFg== 2024-09-21T01:02:06.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64380b65f0 msgr2=0x7f64380b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:06.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64380b65f0 0x7f64380b8b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f645406b2a0 tx=0x7f6440005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 msgr2=0x7f645419c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f645419c6d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6444005940 tx=0x7f64440040c0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 shutdown_connections 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64380b65f0 0x7f64380b8b00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6454109e00 0x7f64541a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64541094b0 0x7f645419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 --2- 172.21.15.39:0/1762780780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6454103b70 0x7f645419c6d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 >> 172.21.15.39:0/1762780780 conn(0x7f64540ff5a0 msgr2=0x7f6454108ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 shutdown_connections 2024-09-21T01:02:06.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:06.570+0000 7f6459a00640 1 -- 172.21.15.39:0/1762780780 wait complete. 2024-09-21T01:02:07.511 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:02:07.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T01:02:07.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T01:02:07.592 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T01:02:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:07 smithi150 ceph-mon[34945]: pgmap v345: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.8 KiB/s wr, 8 op/s 2024-09-21T01:02:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:07 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1762780780' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:02:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:07 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1762780780' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:02:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:07 smithi160 ceph-mon[34696]: pgmap v345: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.8 KiB/s wr, 8 op/s 2024-09-21T01:02:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:07 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1762780780' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:02:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:07 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1762780780' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:02:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:07 smithi039 ceph-mon[30415]: pgmap v345: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.8 KiB/s wr, 8 op/s 2024-09-21T01:02:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:07 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1762780780' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:02:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:07 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1762780780' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:02:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2453393316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd41092d0 msgr2=0x7fcfd41096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2453393316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd41092d0 0x7fcfd41096e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009900 tx=0x7fcfbc02a410 comp rx=0 tx=0).stop 2024-09-21T01:02:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2453393316 shutdown_connections 2024-09-21T01:02:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2453393316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4109c20 0x7fcfd4110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2453393316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd41092d0 0x7fcfd41096e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2453393316 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfd4103990 0x7fcfd4103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2453393316 >> 172.21.15.39:0/2453393316 conn(0x7fcfd40ff440 msgr2=0x7fcfd4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2453393316 shutdown_connections 2024-09-21T01:02:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.692+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2453393316 wait complete. 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 Processor -- start 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 -- start start 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 0x7fcfd4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfd41092d0 0x7fcfd4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4109c20 0x7fcfd4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd41138e0 con 0x7fcfd4103990 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4113760 con 0x7fcfd4109c20 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.693+0000 7fcfd3e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4113a60 con 0x7fcfd41092d0 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 0x7fcfd4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 0x7fcfd4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58872/0 (socket says 172.21.15.39:58872) 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 -- 172.21.15.39:0/2115055961 learned_addr learned my addr 172.21.15.39:0/2115055961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfca000640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfd41092d0 0x7fcfd4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd3400640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4109c20 0x7fcfd4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 -- 172.21.15.39:0/2115055961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfd41092d0 msgr2=0x7fcfd4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfd41092d0 0x7fcfd4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 -- 172.21.15.39:0/2115055961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4109c20 msgr2=0x7fcfd4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4109c20 0x7fcfd4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 -- 172.21.15.39:0/2115055961 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd41a8690 con 0x7fcfd4103990 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.694+0000 7fcfd2a00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 0x7fcfd4075710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400e9f0 tx=0x7fcfc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.695+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400cd80 con 0x7fcfd4103990 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.695+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd41a8980 con 0x7fcfd4103990 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.695+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc40054d0 con 0x7fcfd4103990 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.695+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4010690 con 0x7fcfd4103990 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.695+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd41a8e40 con 0x7fcfd4103990 2024-09-21T01:02:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.696+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd4103da0 con 0x7fcfd4103990 2024-09-21T01:02:08.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.698+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fcfc4004e90 con 0x7fcfd4103990 2024-09-21T01:02:08.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.700+0000 7fcfcbe00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfb80b65f0 0x7fcfb80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:08.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.700+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fcfc40e7d60 con 0x7fcfd4103990 2024-09-21T01:02:08.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.700+0000 7fcfca000640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfb80b65f0 0x7fcfb80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:08.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.701+0000 7fcfca000640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfb80b65f0 0x7fcfb80b8b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009230 tx=0x7fcfbc031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:08.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.704+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcfc40ae000 con 0x7fcfd4103990 2024-09-21T01:02:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.905+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fcfd410b480 con 0x7fcfd4103990 2024-09-21T01:02:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.906+0000 7fcfcbe00640 1 -- 172.21.15.39:0/2115055961 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v55) ==== 86+0+133 (secure 0 0 0) 0x7fcfc40b23e0 con 0x7fcfd4103990 2024-09-21T01:02:08.907 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.908+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfb80b65f0 msgr2=0x7fcfb80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.908+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfb80b65f0 0x7fcfb80b8b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009230 tx=0x7fcfbc031400 comp rx=0 tx=0).stop 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.908+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 msgr2=0x7fcfd4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.908+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 0x7fcfd4075710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400e9f0 tx=0x7fcfc400edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 shutdown_connections 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfb80b65f0 0x7fcfb80b8b00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4109c20 0x7fcfd4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcfd41092d0 0x7fcfd4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 --2- 172.21.15.39:0/2115055961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfd4103990 0x7fcfd4075710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:08.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 >> 172.21.15.39:0/2115055961 conn(0x7fcfd40ff440 msgr2=0x7fcfd4101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:08.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 shutdown_connections 2024-09-21T01:02:08.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:08.909+0000 7fcfd3e00640 1 -- 172.21.15.39:0/2115055961 wait complete. 2024-09-21T01:02:09.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:09 smithi039 ceph-mon[30415]: pgmap v346: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T01:02:09.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]} : dispatch 2024-09-21T01:02:09.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]} : dispatch 2024-09-21T01:02:09.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]} : dispatch 2024-09-21T01:02:09.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]} : dispatch 2024-09-21T01:02:09.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:09 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2115055961' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T01:02:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:09 smithi150 ceph-mon[34945]: pgmap v346: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T01:02:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]} : dispatch 2024-09-21T01:02:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]} : dispatch 2024-09-21T01:02:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]} : dispatch 2024-09-21T01:02:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]} : dispatch 2024-09-21T01:02:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:09 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2115055961' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T01:02:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:09 smithi160 ceph-mon[34696]: pgmap v346: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T01:02:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]} : dispatch 2024-09-21T01:02:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]} : dispatch 2024-09-21T01:02:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]} : dispatch 2024-09-21T01:02:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]} : dispatch 2024-09-21T01:02:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:09 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2115055961' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T01:02:09.863 INFO:teuthology.orchestra.run.smithi039.stdout:[{"version":55,"timestamp":"2024-09-21T01:00:29.669345+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-09-21T01:02:09.864 INFO:tasks.ceph_manager:config epoch is 55 2024-09-21T01:02:09.864 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T01:02:09.864 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T01:02:09.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T01:02:10.241 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:10.651 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-09-21T01:02:10.651 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-21T01:02:10.651 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-09-21T01:02:10.652 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-09-21T01:02:10.652 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:10 smithi039 ceph-mon[30415]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T01:02:10.652 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:10 smithi039 ceph-mon[30415]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T01:02:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-09-21T01:02:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-21T01:02:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-09-21T01:02:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-09-21T01:02:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:10 smithi150 ceph-mon[34945]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T01:02:10.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:10 smithi150 ceph-mon[34945]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T01:02:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-09-21T01:02:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-21T01:02:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-09-21T01:02:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-09-21T01:02:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:10 smithi160 ceph-mon[34696]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T01:02:10.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:10 smithi160 ceph-mon[34696]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 -- 172.21.15.39:0/1476791654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4109df0 msgr2=0x7f3ae4110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 --2- 172.21.15.39:0/1476791654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4109df0 0x7f3ae4110620 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3adc009ca0 tx=0x7f3adc02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 -- 172.21.15.39:0/1476791654 shutdown_connections 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 --2- 172.21.15.39:0/1476791654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4109df0 0x7f3ae4110620 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 --2- 172.21.15.39:0/1476791654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae41094a0 0x7f3ae41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 --2- 172.21.15.39:0/1476791654 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ae4103b60 0x7f3ae4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 -- 172.21.15.39:0/1476791654 >> 172.21.15.39:0/1476791654 conn(0x7f3ae407b850 msgr2=0x7f3ae407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 -- 172.21.15.39:0/1476791654 shutdown_connections 2024-09-21T01:02:11.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.014+0000 7f3aeb000640 1 -- 172.21.15.39:0/1476791654 wait complete. 2024-09-21T01:02:11.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.015+0000 7f3aeb000640 1 Processor -- start 2024-09-21T01:02:11.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 -- start start 2024-09-21T01:02:11.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 0x7f3ae4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:11.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ae41094a0 0x7f3ae4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:11.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae4109df0 0x7f3ae4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:11.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae4113b00 con 0x7f3ae4103b60 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae4113980 con 0x7f3ae4109df0 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3aeb000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae4113c80 con 0x7f3ae41094a0 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3ae3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 0x7f3ae4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3ae3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 0x7f3ae4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58896/0 (socket says 172.21.15.39:58896) 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3ae3e00640 1 -- 172.21.15.39:0/577222268 learned_addr learned my addr 172.21.15.39:0/577222268 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3adb400640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ae41094a0 0x7f3ae4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.016+0000 7f3ae9000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae4109df0 0x7f3ae4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae3e00640 1 -- 172.21.15.39:0/577222268 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ae41094a0 msgr2=0x7f3ae4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae3e00640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ae41094a0 0x7f3ae4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae3e00640 1 -- 172.21.15.39:0/577222268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae4109df0 msgr2=0x7f3ae4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae3e00640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae4109df0 0x7f3ae4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae3e00640 1 -- 172.21.15.39:0/577222268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae41a8710 con 0x7f3ae4103b60 2024-09-21T01:02:11.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae9000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae4109df0 0x7f3ae4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:11.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae3e00640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 0x7f3ae4075710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400e9f0 tx=0x7f3ad400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:11.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.017+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400cd80 con 0x7f3ae4103b60 2024-09-21T01:02:11.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.018+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad40054d0 con 0x7f3ae4103b60 2024-09-21T01:02:11.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.018+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae41a8a00 con 0x7f3ae4103b60 2024-09-21T01:02:11.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.018+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4010690 con 0x7f3ae4103b60 2024-09-21T01:02:11.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.018+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae41a8ef0 con 0x7f3ae4103b60 2024-09-21T01:02:11.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.019+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae4103f70 con 0x7f3ae4103b60 2024-09-21T01:02:11.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.021+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f3ad4004e30 con 0x7f3ae4103b60 2024-09-21T01:02:11.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.023+0000 7f3ae1600640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ac80b65f0 0x7f3ac80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:11.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.023+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3ad40e8060 con 0x7f3ae4103b60 2024-09-21T01:02:11.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.023+0000 7f3adb400640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ac80b65f0 0x7f3ac80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:11.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.024+0000 7f3adb400640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ac80b65f0 0x7f3ac80b8b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3ae41091a0 tx=0x7f3acc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:11.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.025+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3ad4056710 con 0x7f3ae4103b60 2024-09-21T01:02:11.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.273+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3ae4076a10 con 0x7f3ae4103b60 2024-09-21T01:02:11.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.281+0000 7f3ae1600640 1 -- 172.21.15.39:0/577222268 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+296453 (secure 0 0 0) 0x7f3ae4076a10 con 0x7f3ae4103b60 2024-09-21T01:02:11.286 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:11.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ac80b65f0 msgr2=0x7f3ac80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:11.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ac80b65f0 0x7f3ac80b8b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3ae41091a0 tx=0x7f3acc005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:11.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 msgr2=0x7f3ae4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 0x7f3ae4075710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400e9f0 tx=0x7f3ad400edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 shutdown_connections 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ac80b65f0 0x7f3ac80b8b00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ae4109df0 0x7f3ae4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ae41094a0 0x7f3ae4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 --2- 172.21.15.39:0/577222268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ae4103b60 0x7f3ae4075710 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.286+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 >> 172.21.15.39:0/577222268 conn(0x7f3ae407b850 msgr2=0x7f3ae41074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.287+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 shutdown_connections 2024-09-21T01:02:11.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:11.287+0000 7f3aeb000640 1 -- 172.21.15.39:0/577222268 wait complete. 2024-09-21T01:02:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:11 smithi039 ceph-mon[30415]: pgmap v348: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-21T01:02:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:11 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/577222268' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T01:02:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:11 smithi150 ceph-mon[34945]: pgmap v348: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-21T01:02:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:11 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/577222268' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T01:02:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:11 smithi160 ceph-mon[34696]: pgmap v348: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-21T01:02:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:11 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/577222268' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T01:02:12.225 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":21,"flags":0,"active_gid":14223,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2923702906},{"type":"v1","addr":"172.21.15.39:6801","nonce":2923702906}]},"active_addr":"172.21.15.39:6801/2923702906","active_change":"2024-09-21T00:53:07.774427+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24169,"name":"z","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24173,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.39:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1426954663}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3729574193}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":2883154799}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":2085490382}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":645580278}]}]} 2024-09-21T01:02:12.232 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T01:02:12.232 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T01:02:12.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:02:12.619 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:13.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.301+0000 7f50a1800640 1 -- 172.21.15.39:0/1240783358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 msgr2=0x7f509c109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:13.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.301+0000 7f50a1800640 1 --2- 172.21.15.39:0/1240783358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c109700 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5088009900 tx=0x7f508802a410 comp rx=0 tx=0).stop 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 -- 172.21.15.39:0/1240783358 shutdown_connections 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 --2- 172.21.15.39:0/1240783358 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f509c109c40 0x7f509c110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 --2- 172.21.15.39:0/1240783358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c109700 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 --2- 172.21.15.39:0/1240783358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c1039b0 0x7f509c103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 -- 172.21.15.39:0/1240783358 >> 172.21.15.39:0/1240783358 conn(0x7f509c0ff400 msgr2=0x7f509c101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 -- 172.21.15.39:0/1240783358 shutdown_connections 2024-09-21T01:02:13.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.302+0000 7f50a1800640 1 -- 172.21.15.39:0/1240783358 wait complete. 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.303+0000 7f50a1800640 1 Processor -- start 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.303+0000 7f50a1800640 1 -- start start 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.303+0000 7f50a1800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f509c1039b0 0x7f509c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.303+0000 7f50a1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.303+0000 7f50a1800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 0x7f509c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.303+0000 7f50a1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f509c113900 con 0x7f509c1092f0 2024-09-21T01:02:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a1800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f509c113780 con 0x7f509c109c40 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a1800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f509c113a80 con 0x7f509c1039b0 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f509b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 0x7f509c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f509be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f509c1039b0 0x7f509c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f509b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39928/0 (socket says 172.21.15.39:39928) 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f509b400640 1 -- 172.21.15.39:0/419569692 learned_addr learned my addr 172.21.15.39:0/419569692 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 0x7f509c076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46394/0 (socket says 172.21.15.39:46394) 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 -- 172.21.15.39:0/419569692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f509c1039b0 msgr2=0x7f509c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f509c1039b0 0x7f509c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 -- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 msgr2=0x7f509c075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.304+0000 7f50a0e00640 1 -- 172.21.15.39:0/419569692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509c1a8590 con 0x7f509c109c40 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.305+0000 7f50a0e00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 0x7f509c076190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5094009fd0 tx=0x7f509400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.305+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509400ce20 con 0x7f509c109c40 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.305+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509c1a8880 con 0x7f509c109c40 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.305+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5094005020 con 0x7f509c109c40 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.305+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5094010690 con 0x7f509c109c40 2024-09-21T01:02:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.305+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509c1a8e10 con 0x7f509c109c40 2024-09-21T01:02:13.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.306+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509c103dc0 con 0x7f509c109c40 2024-09-21T01:02:13.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.309+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f50940054d0 con 0x7f509c109c40 2024-09-21T01:02:13.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.311+0000 7f5098c00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50800b65f0 0x7f50800b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:13.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.311+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f50940e80b0 con 0x7f509c109c40 2024-09-21T01:02:13.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.311+0000 7f509be00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50800b65f0 0x7f50800b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:13.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.312+0000 7f509be00640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50800b65f0 0x7f50800b8b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f509c077160 tx=0x7f508c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:13.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.312+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f50940ae2e0 con 0x7f509c109c40 2024-09-21T01:02:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.512+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f509c10b4a0 con 0x7f509c109c40 2024-09-21T01:02:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.514+0000 7f5098c00640 1 -- 172.21.15.39:0/419569692 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25331 (secure 0 0 0) 0x7f50940b26c0 con 0x7f509c109c40 2024-09-21T01:02:13.515 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:13.516 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":90,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:02:10.480352+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50800b65f0 msgr2=0x7f50800b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50800b65f0 0x7f50800b8b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f509c077160 tx=0x7f508c005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 msgr2=0x7f509c076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 0x7f509c076190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5094009fd0 tx=0x7f509400ef10 comp rx=0 tx=0).stop 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 shutdown_connections 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f50800b65f0 0x7f50800b8b00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f509c109c40 0x7f509c076190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f509c1092f0 0x7f509c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 --2- 172.21.15.39:0/419569692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f509c1039b0 0x7f509c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:13.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.516+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 >> 172.21.15.39:0/419569692 conn(0x7f509c0ff400 msgr2=0x7f509c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:13.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.517+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 shutdown_connections 2024-09-21T01:02:13.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:13.517+0000 7f50a1800640 1 -- 172.21.15.39:0/419569692 wait complete. 2024-09-21T01:02:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:13 smithi150 ceph-mon[34945]: pgmap v350: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-21T01:02:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:13 smithi160 ceph-mon[34696]: pgmap v350: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-21T01:02:13.891 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:13 smithi039 ceph-mon[30415]: pgmap v350: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-21T01:02:14.293 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T01:02:14.293 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:02:14.677 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:14.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:14 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/419569692' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:14.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:14 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/419569692' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:14.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:14 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/419569692' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:15.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 -- 172.21.15.39:0/3297814669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710103b60 msgr2=0x7f8710103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:15.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 --2- 172.21.15.39:0/3297814669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710103b60 0x7f8710103f70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8700009900 tx=0x7f870002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 -- 172.21.15.39:0/3297814669 shutdown_connections 2024-09-21T01:02:15.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 --2- 172.21.15.39:0/3297814669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710109df0 0x7f8710110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 --2- 172.21.15.39:0/3297814669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 0x7f87101098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 --2- 172.21.15.39:0/3297814669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710103b60 0x7f8710103f70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 -- 172.21.15.39:0/3297814669 >> 172.21.15.39:0/3297814669 conn(0x7f871007b850 msgr2=0x7f871007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:15.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 -- 172.21.15.39:0/3297814669 shutdown_connections 2024-09-21T01:02:15.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.317+0000 7f8717200640 1 -- 172.21.15.39:0/3297814669 wait complete. 2024-09-21T01:02:15.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.319+0000 7f8717200640 1 Processor -- start 2024-09-21T01:02:15.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.319+0000 7f8717200640 1 -- start start 2024-09-21T01:02:15.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.319+0000 7f8717200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 0x7f871019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:15.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8717200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 0x7f871019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:15.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8717200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710109df0 0x7f871019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:15.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8717200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8710113b00 con 0x7f8710109df0 2024-09-21T01:02:15.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8717200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8710113980 con 0x7f8710103b60 2024-09-21T01:02:15.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8717200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8710113c80 con 0x7f87101094a0 2024-09-21T01:02:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8707e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 0x7f871019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 0x7f871019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8715200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710109df0 0x7f871019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8707e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 0x7f871019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:47156/0 (socket says 172.21.15.39:47156) 2024-09-21T01:02:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 0x7f871019a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46398/0 (socket says 172.21.15.39:46398) 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f8707e00640 1 -- 172.21.15.39:0/2523662711 learned_addr learned my addr 172.21.15.39:0/2523662711 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 -- 172.21.15.39:0/2523662711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 msgr2=0x7f871019aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 0x7f871019aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 -- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710109df0 msgr2=0x7f871019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710109df0 0x7f871019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.320+0000 7f870fe00640 1 -- 172.21.15.39:0/2523662711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87101a0420 con 0x7f8710103b60 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.321+0000 7f870fe00640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 0x7f871019a4b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f870002a9e0 tx=0x7f8700003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.321+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870003f070 con 0x7f8710103b60 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.321+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87101a06b0 con 0x7f8710103b60 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.321+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8700031b50 con 0x7f8710103b60 2024-09-21T01:02:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.321+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700035030 con 0x7f8710103b60 2024-09-21T01:02:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.321+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87101a6e70 con 0x7f8710103b60 2024-09-21T01:02:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.322+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8710104de0 con 0x7f8710103b60 2024-09-21T01:02:15.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.325+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f8700042070 con 0x7f8710103b60 2024-09-21T01:02:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.326+0000 7f870d600640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f40b65f0 0x7f86f40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.327+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8700108170 con 0x7f8710103b60 2024-09-21T01:02:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.327+0000 7f8707e00640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f40b65f0 0x7f86f40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.327+0000 7f8707e00640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f40b65f0 0x7f86f40b8b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8710109200 tx=0x7f86f8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:15.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.328+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f87000ce3a0 con 0x7f8710103b60 2024-09-21T01:02:15.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.536+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f871010bbc0 con 0x7f8710103b60 2024-09-21T01:02:15.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.538+0000 7f870d600640 1 -- 172.21.15.39:0/2523662711 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25331 (secure 0 0 0) 0x7f87000d2780 con 0x7f8710103b60 2024-09-21T01:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":90,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:02:10.480352+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f40b65f0 msgr2=0x7f86f40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f40b65f0 0x7f86f40b8b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8710109200 tx=0x7f86f8005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 msgr2=0x7f871019a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 0x7f871019a4b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f870002a9e0 tx=0x7f8700003b90 comp rx=0 tx=0).stop 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 shutdown_connections 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8710109df0 0x7f871019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f87101094a0 0x7f871019aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f86f40b65f0 0x7f86f40b8b00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 --2- 172.21.15.39:0/2523662711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8710103b60 0x7f871019a4b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:15.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.540+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 >> 172.21.15.39:0/2523662711 conn(0x7f871007b850 msgr2=0x7f8710107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:15.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.541+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 shutdown_connections 2024-09-21T01:02:15.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:15.541+0000 7f8717200640 1 -- 172.21.15.39:0/2523662711 wait complete. 2024-09-21T01:02:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:15 smithi150 ceph-mon[34945]: pgmap v351: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-21T01:02:15.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:15 smithi160 ceph-mon[34696]: pgmap v351: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-21T01:02:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:15 smithi039 ceph-mon[30415]: pgmap v351: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-21T01:02:16.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T01:02:16.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T01:02:16.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T01:02:16.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T01:02:16.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T01:02:16.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T01:02:16.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T01:02:16.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T01:02:16.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.8 flush_pg_stats 2024-09-21T01:02:16.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.9 flush_pg_stats 2024-09-21T01:02:16.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.10 flush_pg_stats 2024-09-21T01:02:16.590 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell osd.11 flush_pg_stats 2024-09-21T01:02:16.675 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:16 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2523662711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:16.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:16 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2523662711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:16.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:16 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2523662711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:02:17.557 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:17.640 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:17.641 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:17.658 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:17.695 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:17 smithi150 ceph-mon[34945]: pgmap v352: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-21T01:02:17.845 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:17 smithi039 ceph-mon[30415]: pgmap v352: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-21T01:02:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:17 smithi160 ceph-mon[34696]: pgmap v352: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-21T01:02:18.268 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:18.381 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:18.385 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:18.385 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.447+0000 7fe377c00640 1 -- 172.21.15.39:0/3389382615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3700757b0 msgr2=0x7fe370075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.447+0000 7fe377c00640 1 --2- 172.21.15.39:0/3389382615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3700757b0 0x7fe370075bc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe364009900 tx=0x7fe36402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 -- 172.21.15.39:0/3389382615 shutdown_connections 2024-09-21T01:02:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 --2- 172.21.15.39:0/3389382615 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3701100e0 0x7fe370112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 --2- 172.21.15.39:0/3389382615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe370076100 0x7fe37010fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 --2- 172.21.15.39:0/3389382615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3700757b0 0x7fe370075bc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 -- 172.21.15.39:0/3389382615 >> 172.21.15.39:0/3389382615 conn(0x7fe3700ff590 msgr2=0x7fe3701019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:18.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 -- 172.21.15.39:0/3389382615 shutdown_connections 2024-09-21T01:02:18.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.448+0000 7fe377c00640 1 -- 172.21.15.39:0/3389382615 wait complete. 2024-09-21T01:02:18.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 Processor -- start 2024-09-21T01:02:18.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 -- start start 2024-09-21T01:02:18.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3700757b0 0x7fe37019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:18.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe370076100 0x7fe37019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:18.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3701100e0 0x7fe3701a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:18.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe370115c90 con 0x7fe3701100e0 2024-09-21T01:02:18.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe370115b10 con 0x7fe370076100 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.449+0000 7fe377c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe370115e10 con 0x7fe3700757b0 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3700757b0 0x7fe37019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3701100e0 0x7fe3701a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3700757b0 0x7fe37019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:47178/0 (socket says 172.21.15.39:47178) 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375200640 1 -- 172.21.15.39:0/3875084177 learned_addr learned my addr 172.21.15.39:0/3875084177 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe36fe00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe370076100 0x7fe37019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375c00640 1 -- 172.21.15.39:0/3875084177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3700757b0 msgr2=0x7fe37019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375c00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3700757b0 0x7fe37019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375c00640 1 -- 172.21.15.39:0/3875084177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe370076100 msgr2=0x7fe37019cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375c00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe370076100 0x7fe37019cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.450+0000 7fe375c00640 1 -- 172.21.15.39:0/3875084177 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3701a25b0 con 0x7fe3701100e0 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.451+0000 7fe375c00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3701100e0 0x7fe3701a1e80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe36800da10 tx=0x7fe36800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.451+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36800bd70 con 0x7fe3701100e0 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.451+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe368003e20 con 0x7fe3701100e0 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.451+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3701a28a0 con 0x7fe3701100e0 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.451+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3680106c0 con 0x7fe3701100e0 2024-09-21T01:02:18.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.451+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3701a9000 con 0x7fe3701100e0 2024-09-21T01:02:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.452+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe370063b50 con 0x7fe3701100e0 2024-09-21T01:02:18.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.455+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fe368004270 con 0x7fe3701100e0 2024-09-21T01:02:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.456+0000 7fe36d600640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3580b6670 0x7fe3580b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.456+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe3680e7d30 con 0x7fe3701100e0 2024-09-21T01:02:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.456+0000 7fe375200640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3580b6670 0x7fe3580b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.457+0000 7fe36d600640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] conn(0x7fe3580c1c00 0x7fe3580c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:18.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.457+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 --> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe368021e50 con 0x7fe3580c1c00 2024-09-21T01:02:18.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.457+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe3680ea350 con 0x7fe3701100e0 2024-09-21T01:02:18.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.457+0000 7fe36fe00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] conn(0x7fe3580c1c00 0x7fe3580c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:18.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.457+0000 7fe375200640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3580b6670 0x7fe3580b8b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe364031440 tx=0x7fe3640058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:18.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.458+0000 7fe36fe00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] conn(0x7fe3580c1c00 0x7fe3580c40f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.459+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== osd.0 v2:172.21.15.39:6802/1725836474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe368021e50 con 0x7fe3580c1c00 2024-09-21T01:02:18.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.473+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 --> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe370063fd0 con 0x7fe3580c1c00 2024-09-21T01:02:18.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.474+0000 7fe36d600640 1 -- 172.21.15.39:0/3875084177 <== osd.0 v2:172.21.15.39:6802/1725836474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe370063fd0 con 0x7fe3580c1c00 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.475+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] conn(0x7fe3580c1c00 msgr2=0x7fe3580c40f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.475+0000 7fe377c00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] conn(0x7fe3580c1c00 0x7fe3580c40f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.475+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3580b6670 msgr2=0x7fe3580b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.475+0000 7fe377c00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3580b6670 0x7fe3580b8b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe364031440 tx=0x7fe3640058d0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.475+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3701100e0 msgr2=0x7fe3701a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.475+0000 7fe377c00640 1 --2- 172.21.15.39:0/3875084177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3701100e0 0x7fe3701a1e80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe36800da10 tx=0x7fe36800ded0 comp rx=0 tx=0).stop 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.476+0000 7fe375c00640 1 -- 172.21.15.39:0/3875084177 reap_dead start 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.476+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 shutdown_connections 2024-09-21T01:02:18.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.476+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 >> 172.21.15.39:0/3875084177 conn(0x7fe3700ff590 msgr2=0x7fe370112240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:18.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.476+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 shutdown_connections 2024-09-21T01:02:18.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:18.476+0000 7fe377c00640 1 -- 172.21.15.39:0/3875084177 wait complete. 2024-09-21T01:02:19.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 -- 172.21.15.39:0/2162815957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 msgr2=0x7f6cd010bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 --2- 172.21.15.39:0/2162815957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd010bad0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009900 tx=0x7f6cb802a410 comp rx=0 tx=0).stop 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 -- 172.21.15.39:0/2162815957 shutdown_connections 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 --2- 172.21.15.39:0/2162815957 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cd010c010 0x7f6cd0112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 --2- 172.21.15.39:0/2162815957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd010bad0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 --2- 172.21.15.39:0/2162815957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd0105d80 0x7f6cd0106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 -- 172.21.15.39:0/2162815957 >> 172.21.15.39:0/2162815957 conn(0x7f6cd0101810 msgr2=0x7f6cd0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 -- 172.21.15.39:0/2162815957 shutdown_connections 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.486+0000 7f6cd5400640 1 -- 172.21.15.39:0/2162815957 wait complete. 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.487+0000 7f6cd5400640 1 Processor -- start 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.487+0000 7f6cd5400640 1 -- start start 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.487+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd0105d80 0x7f6cd019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:19.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.487+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cd010c010 0x7f6cd01a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cd5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0115cd0 con 0x7f6cd010b6c0 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cd5400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0115b50 con 0x7f6cd0105d80 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cd5400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0115e50 con 0x7f6cd010c010 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd019cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39964/0 (socket says 172.21.15.39:39964) 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 -- 172.21.15.39:0/3133328155 learned_addr learned my addr 172.21.15.39:0/3133328155 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cce800640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd0105d80 0x7f6cd019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6ccf200640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cd010c010 0x7f6cd01a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 -- 172.21.15.39:0/3133328155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cd010c010 msgr2=0x7f6cd01a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cd010c010 0x7f6cd01a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 -- 172.21.15.39:0/3133328155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd0105d80 msgr2=0x7f6cd019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd0105d80 0x7f6cd019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:19.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6cc7e00640 1 -- 172.21.15.39:0/3133328155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd01a2630 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.488+0000 7f6ccf200640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6cd010c010 0x7f6cd01a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.489+0000 7f6cc7e00640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd019cc60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6cb802a930 tx=0x7f6cb8031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.489+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb803f030 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.489+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb8004490 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.489+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd01a28c0 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.489+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8035070 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.489+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd01a9080 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.490+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6cd0063b50 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.492+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f6cb8042030 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.494+0000 7f6cc7400640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cb40b65f0 0x7f6cb40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.494+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6cb81081d0 con 0x7f6cd010b6c0 2024-09-21T01:02:19.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.494+0000 7f6cce800640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cb40b65f0 0x7f6cb40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.494+0000 7f6cc7400640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] conn(0x7f6cb40c1b80 0x7f6cb40c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.494+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 --> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6cb804eea0 con 0x7f6cb40c1b80 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.494+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6cb8108580 con 0x7f6cd010b6c0 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.495+0000 7f6ccf200640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] conn(0x7f6cb40c1b80 0x7f6cb40c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.495+0000 7f6cce800640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cb40b65f0 0x7f6cb40b8b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009bd0 tx=0x7f6cc0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.495+0000 7f6ccf200640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] conn(0x7f6cb40c1b80 0x7f6cb40c4070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.496+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== osd.6 v2:172.21.15.39:6818/1468848111 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6cb804eea0 con 0x7f6cb40c1b80 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.511+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 --> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6cd0063fd0 con 0x7f6cb40c1b80 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.511+0000 7f6cc7400640 1 -- 172.21.15.39:0/3133328155 <== osd.6 v2:172.21.15.39:6818/1468848111 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6cd0063fd0 con 0x7f6cb40c1b80 2024-09-21T01:02:19.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.511+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] conn(0x7f6cb40c1b80 msgr2=0x7f6cb40c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.511+0000 7f6cd5400640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] conn(0x7f6cb40c1b80 0x7f6cb40c4070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cb40b65f0 msgr2=0x7f6cb40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cb40b65f0 0x7f6cb40b8b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009bd0 tx=0x7f6cc0005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 msgr2=0x7f6cd019cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 --2- 172.21.15.39:0/3133328155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cd010b6c0 0x7f6cd019cc60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6cb802a930 tx=0x7f6cb8031a20 comp rx=0 tx=0).stop 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6ccf200640 1 -- 172.21.15.39:0/3133328155 reap_dead start 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 shutdown_connections 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 >> 172.21.15.39:0/3133328155 conn(0x7f6cd0101810 msgr2=0x7f6cd0103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 shutdown_connections 2024-09-21T01:02:19.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:19.512+0000 7f6cd5400640 1 -- 172.21.15.39:0/3133328155 wait complete. 2024-09-21T01:02:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:19 smithi150 ceph-mon[34945]: pgmap v353: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 243 B/s wr, 7 op/s 2024-09-21T01:02:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:19 smithi160 ceph-mon[34696]: pgmap v353: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 243 B/s wr, 7 op/s 2024-09-21T01:02:19.890 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:19 smithi039 ceph-mon[30415]: pgmap v353: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 243 B/s wr, 7 op/s 2024-09-21T01:02:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 -- 172.21.15.39:0/3638214484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f80760a0 msgr2=0x7fa3f810fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 --2- 172.21.15.39:0/3638214484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f80760a0 0x7fa3f810fbe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009900 tx=0x7fa3e002a410 comp rx=0 tx=0).stop 2024-09-21T01:02:20.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 -- 172.21.15.39:0/3638214484 shutdown_connections 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 --2- 172.21.15.39:0/3638214484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f8110120 0x7fa3f8112540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 --2- 172.21.15.39:0/3638214484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f80760a0 0x7fa3f810fbe0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 --2- 172.21.15.39:0/3638214484 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa3f8075750 0x7fa3f8075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 -- 172.21.15.39:0/3638214484 >> 172.21.15.39:0/3638214484 conn(0x7fa3f807b850 msgr2=0x7fa3f807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 -- 172.21.15.39:0/3638214484 shutdown_connections 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.154+0000 7fa3fe600640 1 -- 172.21.15.39:0/3638214484 wait complete. 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.155+0000 7fa3fe600640 1 Processor -- start 2024-09-21T01:02:20.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.155+0000 7fa3fe600640 1 -- start start 2024-09-21T01:02:20.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3fe600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f8075750 0x7fa3f819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:20.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3fe600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa3f80760a0 0x7fa3f819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:20.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3fe600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f8110120 0x7fa3f81a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:20.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3fe600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f8115c90 con 0x7fa3f8110120 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3fe600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f8115b10 con 0x7fa3f8075750 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3fe600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f8115e10 con 0x7fa3f80760a0 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f8110120 0x7fa3f81a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3f7400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f8075750 0x7fa3f819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f8110120 0x7fa3f81a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39970/0 (socket says 172.21.15.39:39970) 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3efe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa3f80760a0 0x7fa3f819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:20.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.156+0000 7fa3f7e00640 1 -- 172.21.15.39:0/2904716415 learned_addr learned my addr 172.21.15.39:0/2904716415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:20.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7e00640 1 -- 172.21.15.39:0/2904716415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa3f80760a0 msgr2=0x7fa3f819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7e00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa3f80760a0 0x7fa3f819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7e00640 1 -- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f8075750 msgr2=0x7fa3f819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7e00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f8075750 0x7fa3f819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7e00640 1 -- 172.21.15.39:0/2904716415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f81a2610 con 0x7fa3f8110120 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7400640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f8075750 0x7fa3f819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f7e00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f8110120 0x7fa3f81a1ee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000eb10 tx=0x7fa3f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0004b20 con 0x7fa3f8110120 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f0004cc0 con 0x7fa3f8110120 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f81a2900 con 0x7fa3f8110120 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0010670 con 0x7fa3f8110120 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.157+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f81a9060 con 0x7fa3f8110120 2024-09-21T01:02:20.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.158+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa3f8063b50 con 0x7fa3f8110120 2024-09-21T01:02:20.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.161+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fa3f0005000 con 0x7fa3f8110120 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.162+0000 7fa3f4c00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3dc0b65f0 0x7fa3dc0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa3f00e82b0 con 0x7fa3f8110120 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3f7400640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3dc0b65f0 0x7fa3dc0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3f4c00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] conn(0x7fa3dc0c1b80 0x7fa3dc0c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 --> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3f0015e80 con 0x7fa3dc0c1b80 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fa3f00e8700 con 0x7fa3f8110120 2024-09-21T01:02:20.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] conn(0x7fa3dc0c1b80 0x7fa3dc0c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:20.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.163+0000 7fa3f7400640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3dc0b65f0 0x7fa3dc0b8b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009fc0 tx=0x7fa3e8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:20.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.164+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] conn(0x7fa3dc0c1b80 0x7fa3dc0c4070 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:20.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.165+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== osd.10 v2:172.21.15.150:6824/2296139693 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3f0015e80 con 0x7fa3dc0c1b80 2024-09-21T01:02:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.179+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 --> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3f8064000 con 0x7fa3dc0c1b80 2024-09-21T01:02:20.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.180+0000 7fa3f4c00640 1 -- 172.21.15.39:0/2904716415 <== osd.10 v2:172.21.15.150:6824/2296139693 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa3f8064000 con 0x7fa3dc0c1b80 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] conn(0x7fa3dc0c1b80 msgr2=0x7fa3dc0c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] conn(0x7fa3dc0c1b80 0x7fa3dc0c4070 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3dc0b65f0 msgr2=0x7fa3dc0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3dc0b65f0 0x7fa3dc0b8b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009fc0 tx=0x7fa3e8005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f8110120 msgr2=0x7fa3f81a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 --2- 172.21.15.39:0/2904716415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f8110120 0x7fa3f81a1ee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000eb10 tx=0x7fa3f000c2d0 comp rx=0 tx=0).stop 2024-09-21T01:02:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3f7e00640 1 -- 172.21.15.39:0/2904716415 reap_dead start 2024-09-21T01:02:20.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 shutdown_connections 2024-09-21T01:02:20.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 >> 172.21.15.39:0/2904716415 conn(0x7fa3f807b850 msgr2=0x7fa3f8101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:20.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 shutdown_connections 2024-09-21T01:02:20.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:20.181+0000 7fa3fe600640 1 -- 172.21.15.39:0/2904716415 wait complete. 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 -- 172.21.15.39:0/3969661750 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c1049d0 msgr2=0x7f613c104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 --2- 172.21.15.39:0/3969661750 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c1049d0 0x7f613c104de0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f612c009900 tx=0x7f612c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 -- 172.21.15.39:0/3969661750 shutdown_connections 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 --2- 172.21.15.39:0/3969661750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f613c10de30 0x7f613c110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 --2- 172.21.15.39:0/3969661750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105320 0x7f613c10d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 --2- 172.21.15.39:0/3969661750 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c1049d0 0x7f613c104de0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 -- 172.21.15.39:0/3969661750 >> 172.21.15.39:0/3969661750 conn(0x7f613c079740 msgr2=0x7f613c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 -- 172.21.15.39:0/3969661750 shutdown_connections 2024-09-21T01:02:21.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.367+0000 7f6142e00640 1 -- 172.21.15.39:0/3969661750 wait complete. 2024-09-21T01:02:21.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.368+0000 7f6142e00640 1 Processor -- start 2024-09-21T01:02:21.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 -- start start 2024-09-21T01:02:21.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f613c1049d0 0x7f613c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105320 0x7f613c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c10de30 0x7f613c1a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f613c113a90 con 0x7f613c1049d0 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f613c113910 con 0x7f613c105320 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f6142e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f613c113c10 con 0x7f613c10de30 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.369+0000 7f613be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f613c1049d0 0x7f613c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105320 0x7f613c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f613c1049d0 0x7f613c19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40022/0 (socket says 172.21.15.39:40022) 2024-09-21T01:02:21.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613be00640 1 -- 172.21.15.39:0/880372582 learned_addr learned my addr 172.21.15.39:0/880372582 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:21.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f6140e00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c10de30 0x7f613c1a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:21.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 -- 172.21.15.39:0/880372582 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c10de30 msgr2=0x7f613c1a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:21.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f613c10de30 0x7f613c1a1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 -- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f613c1049d0 msgr2=0x7f613c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:21.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f613c1049d0 0x7f613c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 -- 172.21.15.39:0/880372582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c1a26f0 con 0x7f613c105320 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.370+0000 7f613b400640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105320 0x7f613c19cd20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f613c06b950 tx=0x7f612800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.371+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612800cc40 con 0x7f613c105320 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.371+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c1a29e0 con 0x7f613c105320 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.371+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612800cde0 con 0x7f613c105320 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.371+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128018840 con 0x7f613c105320 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.371+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c1a9140 con 0x7f613c105320 2024-09-21T01:02:21.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.372+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f613c10d770 con 0x7f613c105320 2024-09-21T01:02:21.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.375+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f61280189e0 con 0x7f613c105320 2024-09-21T01:02:21.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.377+0000 7f6138c00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61200b6670 0x7f61200b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:21.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.377+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f61280e80b0 con 0x7f613c105320 2024-09-21T01:02:21.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.377+0000 7f613be00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61200b6670 0x7f61200b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:21.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.377+0000 7f6138c00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] conn(0x7f61200c1c00 0x7f61200c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:21.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.377+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 --> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61200c47c0 con 0x7f61200c1c00 2024-09-21T01:02:21.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.377+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f61280e8460 con 0x7f613c105320 2024-09-21T01:02:21.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.378+0000 7f6140e00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] conn(0x7f61200c1c00 0x7f61200c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:21.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.378+0000 7f613be00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61200b6670 0x7f61200b8b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f612c005f70 tx=0x7f612c009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:21.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.378+0000 7f6140e00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] conn(0x7f61200c1c00 0x7f61200c40f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:21.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.379+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== osd.3 v2:172.21.15.39:6810/678333449 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61200c47c0 con 0x7f61200c1c00 2024-09-21T01:02:21.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.394+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 --> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f613c06b390 con 0x7f61200c1c00 2024-09-21T01:02:21.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.394+0000 7f6138c00640 1 -- 172.21.15.39:0/880372582 <== osd.3 v2:172.21.15.39:6810/678333449 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f613c06b390 con 0x7f61200c1c00 2024-09-21T01:02:21.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] conn(0x7f61200c1c00 msgr2=0x7f61200c40f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:21.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] conn(0x7f61200c1c00 0x7f61200c40f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61200b6670 msgr2=0x7f61200b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61200b6670 0x7f61200b8b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f612c005f70 tx=0x7f612c009560 comp rx=0 tx=0).stop 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105320 msgr2=0x7f613c19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 --2- 172.21.15.39:0/880372582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105320 0x7f613c19cd20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f613c06b950 tx=0x7f612800ee50 comp rx=0 tx=0).stop 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6140e00640 1 -- 172.21.15.39:0/880372582 reap_dead start 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 shutdown_connections 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 >> 172.21.15.39:0/880372582 conn(0x7f613c079740 msgr2=0x7f613c10cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 shutdown_connections 2024-09-21T01:02:21.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.395+0000 7f6142e00640 1 -- 172.21.15.39:0/880372582 wait complete. 2024-09-21T01:02:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:21 smithi150 ceph-mon[34945]: pgmap v354: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 204 B/s wr, 6 op/s 2024-09-21T01:02:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:21 smithi160 ceph-mon[34696]: pgmap v354: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 204 B/s wr, 6 op/s 2024-09-21T01:02:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:21 smithi039 ceph-mon[30415]: pgmap v354: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 204 B/s wr, 6 op/s 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 -- 172.21.15.39:0/675676854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc10ff10 msgr2=0x7fb1dc112350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/675676854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc10ff10 0x7fb1dc112350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009ca0 tx=0x7fb1d402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 -- 172.21.15.39:0/675676854 shutdown_connections 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/675676854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc10ff10 0x7fb1dc112350 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/675676854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1dc076100 0x7fb1dc10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/675676854 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1dc0757b0 0x7fb1dc075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 -- 172.21.15.39:0/675676854 >> 172.21.15.39:0/675676854 conn(0x7fb1dc0ff440 msgr2=0x7fb1dc101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 -- 172.21.15.39:0/675676854 shutdown_connections 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.489+0000 7fb1e0e00640 1 -- 172.21.15.39:0/675676854 wait complete. 2024-09-21T01:02:22.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.490+0000 7fb1e0e00640 1 Processor -- start 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.490+0000 7fb1e0e00640 1 -- start start 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1e0e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc0757b0 0x7fb1dc19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1e0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1dc076100 0x7fb1dc19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1e0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1dc10ff10 0x7fb1dc1a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1e0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc115a80 con 0x7fb1dc10ff10 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1e0e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc115900 con 0x7fb1dc0757b0 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1e0e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc115c00 con 0x7fb1dc076100 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1db400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc0757b0 0x7fb1dc19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1db400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc0757b0 0x7fb1dc19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46494/0 (socket says 172.21.15.39:46494) 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1db400640 1 -- 172.21.15.39:0/1454188258 learned_addr learned my addr 172.21.15.39:0/1454188258 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1d3e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1dc076100 0x7fb1dc19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.491+0000 7fb1dbe00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1dc10ff10 0x7fb1dc1a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d3e00640 1 -- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc0757b0 msgr2=0x7fb1dc19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d3e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1dc0757b0 0x7fb1dc19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d3e00640 1 -- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1dc10ff10 msgr2=0x7fb1dc1a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d3e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1dc10ff10 0x7fb1dc1a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d3e00640 1 -- 172.21.15.39:0/1454188258 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc1a2380 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d3e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1dc076100 0x7fb1dc19c9b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400e9c0 tx=0x7fb1c400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c400cd80 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc1a2670 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.492+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c40108e0 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.493+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40187f0 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.493+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc1a8dd0 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.494+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1dc10a480 con 0x7fb1dc076100 2024-09-21T01:02:22.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.496+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fb1c401f020 con 0x7fb1dc076100 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.499+0000 7fb1d8c00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb1c00b65f0 0x7fb1c00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.499+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb1c40e8220 con 0x7fb1dc076100 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.499+0000 7fb1db400640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb1c00b65f0 0x7fb1c00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.499+0000 7fb1d8c00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] conn(0x7fb1c00c1b80 0x7fb1c00c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.499+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 --> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1c4015e80 con 0x7fb1c00c1b80 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.499+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb1c40e85d0 con 0x7fb1dc076100 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.500+0000 7fb1dbe00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] conn(0x7fb1c00c1b80 0x7fb1c00c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.500+0000 7fb1db400640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb1c00b65f0 0x7fb1c00b8b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009c90 tx=0x7fb1cc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.500+0000 7fb1dbe00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] conn(0x7fb1c00c1b80 0x7fb1c00c4070 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.501+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== osd.7 v2:172.21.15.150:6816/3982567281 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb1c4015e80 con 0x7fb1c00c1b80 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.516+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 --> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb1dc076cc0 con 0x7fb1c00c1b80 2024-09-21T01:02:22.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.516+0000 7fb1d8c00640 1 -- 172.21.15.39:0/1454188258 <== osd.7 v2:172.21.15.150:6816/3982567281 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb1dc076cc0 con 0x7fb1c00c1b80 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] conn(0x7fb1c00c1b80 msgr2=0x7fb1c00c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] conn(0x7fb1c00c1b80 0x7fb1c00c4070 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb1c00b65f0 msgr2=0x7fb1c00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb1c00b65f0 0x7fb1c00b8b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009c90 tx=0x7fb1cc005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1dc076100 msgr2=0x7fb1dc19c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 --2- 172.21.15.39:0/1454188258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1dc076100 0x7fb1dc19c9b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400e9c0 tx=0x7fb1c400ee80 comp rx=0 tx=0).stop 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1dbe00640 1 -- 172.21.15.39:0/1454188258 reap_dead start 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 shutdown_connections 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 >> 172.21.15.39:0/1454188258 conn(0x7fb1dc0ff440 msgr2=0x7fb1dc109a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 shutdown_connections 2024-09-21T01:02:22.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.517+0000 7fb1e0e00640 1 -- 172.21.15.39:0/1454188258 wait complete. 2024-09-21T01:02:22.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 -- 172.21.15.39:0/4036939947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4103990 msgr2=0x7fa7b4103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 --2- 172.21.15.39:0/4036939947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4103990 0x7fa7b4103da0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009900 tx=0x7fa7a402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 -- 172.21.15.39:0/4036939947 shutdown_connections 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 --2- 172.21.15.39:0/4036939947 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa7b4109c20 0x7fa7b4110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 --2- 172.21.15.39:0/4036939947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b41092d0 0x7fa7b41096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 --2- 172.21.15.39:0/4036939947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4103990 0x7fa7b4103da0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 -- 172.21.15.39:0/4036939947 >> 172.21.15.39:0/4036939947 conn(0x7fa7b40ff440 msgr2=0x7fa7b4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 -- 172.21.15.39:0/4036939947 shutdown_connections 2024-09-21T01:02:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.806+0000 7fa7b3200640 1 -- 172.21.15.39:0/4036939947 wait complete. 2024-09-21T01:02:22.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.807+0000 7fa7b3200640 1 Processor -- start 2024-09-21T01:02:22.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.807+0000 7fa7b3200640 1 -- start start 2024-09-21T01:02:22.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b3200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b4103990 0x7fa7b4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b3200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa7b41092d0 0x7fa7b4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4109c20 0x7fa7b4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b41138e0 con 0x7fa7b4109c20 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b3200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4113760 con 0x7fa7b4103990 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b3200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4113a60 con 0x7fa7b41092d0 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b4103990 0x7fa7b4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4109c20 0x7fa7b4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b4103990 0x7fa7b4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46524/0 (socket says 172.21.15.39:46524) 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa7b41092d0 0x7fa7b4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b1e00640 1 -- 172.21.15.39:0/1672636665 learned_addr learned my addr 172.21.15.39:0/1672636665 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.808+0000 7fa7b1e00640 1 -- 172.21.15.39:0/1672636665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa7b41092d0 msgr2=0x7fa7b4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b1e00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa7b41092d0 0x7fa7b4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b1e00640 1 -- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4109c20 msgr2=0x7fa7b4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b1e00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4109c20 0x7fa7b4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b1e00640 1 -- 172.21.15.39:0/1672636665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b41a8670 con 0x7fa7b4103990 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b1e00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b4103990 0x7fa7b4075710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4005940 tx=0x7fa7a40040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a403f030 con 0x7fa7b4103990 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7a40044c0 con 0x7fa7b4103990 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b41a8900 con 0x7fa7b4103990 2024-09-21T01:02:22.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4035070 con 0x7fa7b4103990 2024-09-21T01:02:22.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.809+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b41a8de0 con 0x7fa7b4103990 2024-09-21T01:02:22.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.810+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa7b4063a90 con 0x7fa7b4103990 2024-09-21T01:02:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.813+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fa7a4042030 con 0x7fa7b4103990 2024-09-21T01:02:22.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.814+0000 7fa7aaa00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7980b65f0 0x7fa7980b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa7a4108410 con 0x7fa7b4103990 2024-09-21T01:02:22.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7abe00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7980b65f0 0x7fa7980b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7aaa00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] conn(0x7fa7980c1b80 0x7fa7980c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:22.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 --> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7980c4740 con 0x7fa7980c1b80 2024-09-21T01:02:22.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fa7a41087c0 con 0x7fa7b4103990 2024-09-21T01:02:22.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7b2800640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] conn(0x7fa7980c1b80 0x7fa7980c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:22.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.815+0000 7fa7abe00640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7980b65f0 0x7fa7980b8b00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4102b40 tx=0x7fa79c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:22.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.816+0000 7fa7b2800640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] conn(0x7fa7980c1b80 0x7fa7980c4070 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:22.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.817+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== osd.9 v2:172.21.15.39:6826/4176461767 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7980c4740 con 0x7fa7980c1b80 2024-09-21T01:02:22.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.831+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 --> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7b4063f10 con 0x7fa7980c1b80 2024-09-21T01:02:22.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7aaa00640 1 -- 172.21.15.39:0/1672636665 <== osd.9 v2:172.21.15.39:6826/4176461767 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa7b4063f10 con 0x7fa7980c1b80 2024-09-21T01:02:22.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] conn(0x7fa7980c1b80 msgr2=0x7fa7980c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] conn(0x7fa7980c1b80 0x7fa7980c4070 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7980b65f0 msgr2=0x7fa7980b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa7980b65f0 0x7fa7980b8b00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4102b40 tx=0x7fa79c005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:22.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b4103990 msgr2=0x7fa7b4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:22.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 --2- 172.21.15.39:0/1672636665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b4103990 0x7fa7b4075710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4005940 tx=0x7fa7a40040c0 comp rx=0 tx=0).stop 2024-09-21T01:02:22.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b2800640 1 -- 172.21.15.39:0/1672636665 reap_dead start 2024-09-21T01:02:22.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 shutdown_connections 2024-09-21T01:02:22.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.832+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 >> 172.21.15.39:0/1672636665 conn(0x7fa7b40ff440 msgr2=0x7fa7b4108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:22.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.833+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 shutdown_connections 2024-09-21T01:02:22.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:22.833+0000 7fa7b3200640 1 -- 172.21.15.39:0/1672636665 wait complete. 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 -- 172.21.15.39:0/2713608002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65741092d0 msgr2=0x7f65741096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 --2- 172.21.15.39:0/2713608002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65741092d0 0x7f65741096e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6560009900 tx=0x7f656002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 -- 172.21.15.39:0/2713608002 shutdown_connections 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 --2- 172.21.15.39:0/2713608002 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6574109c20 0x7f6574110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 --2- 172.21.15.39:0/2713608002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65741092d0 0x7f65741096e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 --2- 172.21.15.39:0/2713608002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6574103990 0x7f6574103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 -- 172.21.15.39:0/2713608002 >> 172.21.15.39:0/2713608002 conn(0x7f65740ff3e0 msgr2=0x7f6574101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 -- 172.21.15.39:0/2713608002 shutdown_connections 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.079+0000 7f6579600640 1 -- 172.21.15.39:0/2713608002 wait complete. 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.080+0000 7f6579600640 1 Processor -- start 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.080+0000 7f6579600640 1 -- start start 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6579600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6574103990 0x7f657419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6579600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65741092d0 0x7f657419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6579600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6574109c20 0x7f657419fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6579600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6574113930 con 0x7f6574103990 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6579600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65741137b0 con 0x7f65741092d0 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6579600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6574113ab0 con 0x7f6574109c20 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6573e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6574103990 0x7f657419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:23.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6573400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65741092d0 0x7f657419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6573400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65741092d0 0x7f657419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:40494/0 (socket says 172.21.15.39:40494) 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6573e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6574103990 0x7f657419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43280/0 (socket says 172.21.15.39:43280) 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6573400640 1 -- 172.21.15.39:0/704697922 learned_addr learned my addr 172.21.15.39:0/704697922 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.081+0000 7f6578c00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6574109c20 0x7f657419fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6573e00640 1 -- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6574109c20 msgr2=0x7f657419fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6573e00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6574109c20 0x7f657419fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6573e00640 1 -- 172.21.15.39:0/704697922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65741092d0 msgr2=0x7f657419a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6573e00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65741092d0 0x7f657419a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6573e00640 1 -- 172.21.15.39:0/704697922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65741a0270 con 0x7f6574103990 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6573e00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6574103990 0x7f657419a300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f656400e9f0 tx=0x7f656400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400ccc0 con 0x7f6574103990 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65640035f0 con 0x7f6574103990 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65741a0560 con 0x7f6574103990 2024-09-21T01:02:23.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400ce60 con 0x7f6574103990 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.082+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65741a6cc0 con 0x7f6574103990 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.083+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6574063b50 con 0x7f6574103990 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.086+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f65640037a0 con 0x7f6574103990 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.087+0000 7f6570c00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65580b65f0 0x7f65580b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.087+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f65640e7ff0 con 0x7f6574103990 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.087+0000 7f6573400640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65580b65f0 0x7f65580b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.088+0000 7f6570c00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] conn(0x7f65580c1b80 0x7f65580c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.088+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 --> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65580c4740 con 0x7f65580c1b80 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.088+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f65640e8420 con 0x7f6574103990 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.088+0000 7f6578c00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] conn(0x7f65580c1b80 0x7f65580c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.088+0000 7f6573400640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65580b65f0 0x7f65580b8b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6560006010 tx=0x7f6560005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.089+0000 7f6578c00640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] conn(0x7f65580c1b80 0x7f65580c4070 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:23.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.090+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== osd.2 v2:172.21.15.160:6800/4131806045 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65580c4740 con 0x7f65580c1b80 2024-09-21T01:02:23.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.104+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 --> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6574063fd0 con 0x7f65580c1b80 2024-09-21T01:02:23.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6570c00640 1 -- 172.21.15.39:0/704697922 <== osd.2 v2:172.21.15.160:6800/4131806045 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6574063fd0 con 0x7f65580c1b80 2024-09-21T01:02:23.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] conn(0x7f65580c1b80 msgr2=0x7f65580c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:23.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6579600640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] conn(0x7f65580c1b80 0x7f65580c4070 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65580b65f0 msgr2=0x7f65580b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6579600640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65580b65f0 0x7f65580b8b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6560006010 tx=0x7f6560005a20 comp rx=0 tx=0).stop 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6574103990 msgr2=0x7f657419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6579600640 1 --2- 172.21.15.39:0/704697922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6574103990 0x7f657419a300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f656400e9f0 tx=0x7f656400edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.105+0000 7f6578c00640 1 -- 172.21.15.39:0/704697922 reap_dead start 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.106+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 shutdown_connections 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.106+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 >> 172.21.15.39:0/704697922 conn(0x7f65740ff3e0 msgr2=0x7f6574101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.106+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 shutdown_connections 2024-09-21T01:02:23.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.106+0000 7f6579600640 1 -- 172.21.15.39:0/704697922 wait complete. 2024-09-21T01:02:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:23 smithi160 ceph-mon[34696]: pgmap v355: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 179 B/s wr, 6 op/s 2024-09-21T01:02:23.890 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:23 smithi039 ceph-mon[30415]: pgmap v355: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 179 B/s wr, 6 op/s 2024-09-21T01:02:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:23 smithi150 ceph-mon[34945]: pgmap v355: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 179 B/s wr, 6 op/s 2024-09-21T01:02:24.382 INFO:teuthology.orchestra.run.smithi039.stdout:81604378710 2024-09-21T01:02:24.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T01:02:25.701 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:25 smithi039 ceph-mon[30415]: pgmap v356: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 -- 172.21.15.39:0/2435059481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5681092d0 msgr2=0x7fd5681096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 --2- 172.21.15.39:0/2435059481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5681092d0 0x7fd5681096e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd554009900 tx=0x7fd55402a410 comp rx=0 tx=0).stop 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 -- 172.21.15.39:0/2435059481 shutdown_connections 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 --2- 172.21.15.39:0/2435059481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd568109c20 0x7fd568110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 --2- 172.21.15.39:0/2435059481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5681092d0 0x7fd5681096e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 --2- 172.21.15.39:0/2435059481 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd568103990 0x7fd568103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 -- 172.21.15.39:0/2435059481 >> 172.21.15.39:0/2435059481 conn(0x7fd5680ff3e0 msgr2=0x7fd568101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 -- 172.21.15.39:0/2435059481 shutdown_connections 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.757+0000 7fd56ee00640 1 -- 172.21.15.39:0/2435059481 wait complete. 2024-09-21T01:02:25.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.758+0000 7fd56ee00640 1 Processor -- start 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.758+0000 7fd56ee00640 1 -- start start 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56ee00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd568103990 0x7fd56819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56ee00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5681092d0 0x7fd56819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd568109c20 0x7fd56819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd568113930 con 0x7fd568109c20 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56ee00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5681137b0 con 0x7fd568103990 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56ee00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd568113ab0 con 0x7fd5681092d0 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5681092d0 0x7fd56819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5681092d0 0x7fd56819a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37948/0 (socket says 172.21.15.39:37948) 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd568109c20 0x7fd56819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.759+0000 7fd56d000640 1 -- 172.21.15.39:0/3040489253 learned_addr learned my addr 172.21.15.39:0/3040489253 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56da00640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd568103990 0x7fd56819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:25.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56d000640 1 -- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd568103990 msgr2=0x7fd56819a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56d000640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd568103990 0x7fd56819a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56d000640 1 -- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd568109c20 msgr2=0x7fd56819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56d000640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd568109c20 0x7fd56819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56d000640 1 -- 172.21.15.39:0/3040489253 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5681a0270 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56d000640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5681092d0 0x7fd56819a8a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd554005ab0 tx=0x7fd554003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55403f070 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd554031b50 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554035030 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.760+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5681a0500 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.761+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5681a6cc0 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.761+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd568063b50 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.764+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fd554042070 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.766+0000 7fd566000640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5500b65f0 0x7fd5500b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.766+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd554108420 con 0x7fd5681092d0 2024-09-21T01:02:25.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.766+0000 7fd56da00640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5500b65f0 0x7fd5500b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.766+0000 7fd566000640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] conn(0x7fd5500c1b80 0x7fd5500c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.766+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 --> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd554009080 con 0x7fd5500c1b80 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.766+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd5541087d0 con 0x7fd5681092d0 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.767+0000 7fd56da00640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5500b65f0 0x7fd5500b8b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009b60 tx=0x7fd55c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.767+0000 7fd56e400640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] conn(0x7fd5500c1b80 0x7fd5500c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.767+0000 7fd56e400640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] conn(0x7fd5500c1b80 0x7fd5500c4070 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:25.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.768+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== osd.4 v2:172.21.15.150:6808/1632873600 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd554009080 con 0x7fd5500c1b80 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.783+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 --> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd568063fd0 con 0x7fd5500c1b80 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd566000640 1 -- 172.21.15.39:0/3040489253 <== osd.4 v2:172.21.15.150:6808/1632873600 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd568063fd0 con 0x7fd5500c1b80 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] conn(0x7fd5500c1b80 msgr2=0x7fd5500c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] conn(0x7fd5500c1b80 0x7fd5500c4070 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5500b65f0 msgr2=0x7fd5500b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5500b65f0 0x7fd5500b8b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009b60 tx=0x7fd55c005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5681092d0 msgr2=0x7fd56819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 --2- 172.21.15.39:0/3040489253 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd5681092d0 0x7fd56819a8a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd554005ab0 tx=0x7fd554003b90 comp rx=0 tx=0).stop 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56e400640 1 -- 172.21.15.39:0/3040489253 reap_dead start 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 shutdown_connections 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 >> 172.21.15.39:0/3040489253 conn(0x7fd5680ff3e0 msgr2=0x7fd568101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:25.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 shutdown_connections 2024-09-21T01:02:25.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:25.784+0000 7fd56ee00640 1 -- 172.21.15.39:0/3040489253 wait complete. 2024-09-21T01:02:25.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:25 smithi160 ceph-mon[34696]: pgmap v356: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-21T01:02:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:25 smithi150 ceph-mon[34945]: pgmap v356: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-21T01:02:26.356 INFO:teuthology.orchestra.run.smithi039.stdout:111669149775 2024-09-21T01:02:26.356 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T01:02:26.703 INFO:teuthology.orchestra.run.smithi039.stdout:206158430264 2024-09-21T01:02:26.703 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T01:02:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:38654705760 2024-09-21T01:02:26.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T01:02:27.447 INFO:teuthology.orchestra.run.smithi039.stdout:184683593790 2024-09-21T01:02:27.448 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T01:02:27.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 -- 172.21.15.39:0/3076085722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e810b4f0 msgr2=0x7fd9e810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:27.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 --2- 172.21.15.39:0/3076085722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e810b4f0 0x7fd9e810b900 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009900 tx=0x7fd9d002a410 comp rx=0 tx=0).stop 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 -- 172.21.15.39:0/3076085722 shutdown_connections 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 --2- 172.21.15.39:0/3076085722 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9e810be40 0x7fd9e8112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 --2- 172.21.15.39:0/3076085722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e810b4f0 0x7fd9e810b900 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 --2- 172.21.15.39:0/3076085722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9e8105bb0 0x7fd9e8105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 -- 172.21.15.39:0/3076085722 >> 172.21.15.39:0/3076085722 conn(0x7fd9e8101680 msgr2=0x7fd9e8103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 -- 172.21.15.39:0/3076085722 shutdown_connections 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.961+0000 7fd9e7200640 1 -- 172.21.15.39:0/3076085722 wait complete. 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.962+0000 7fd9e7200640 1 Processor -- start 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.962+0000 7fd9e7200640 1 -- start start 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.962+0000 7fd9e7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e8105bb0 0x7fd9e819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.962+0000 7fd9e7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9e810b4f0 0x7fd9e819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.962+0000 7fd9e7200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9e810be40 0x7fd9e81a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:27.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e8115b50 con 0x7fd9e8105bb0 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e7200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e81159d0 con 0x7fd9e810be40 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e7200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e8115cd0 con 0x7fd9e810b4f0 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e8105bb0 0x7fd9e819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e8105bb0 0x7fd9e819c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43298/0 (socket says 172.21.15.39:43298) 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 -- 172.21.15.39:0/2204410944 learned_addr learned my addr 172.21.15.39:0/2204410944 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9dd400640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9e810b4f0 0x7fd9e819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e6800640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9e810be40 0x7fd9e81a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 -- 172.21.15.39:0/2204410944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9e810b4f0 msgr2=0x7fd9e819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9e810b4f0 0x7fd9e819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 -- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9e810be40 msgr2=0x7fd9e81a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9e810be40 0x7fd9e81a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e5e00640 1 -- 172.21.15.39:0/2204410944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e81a2590 con 0x7fd9e8105bb0 2024-09-21T01:02:27.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.963+0000 7fd9e6800640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9e810be40 0x7fd9e81a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.964+0000 7fd9e5e00640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e8105bb0 0x7fd9e819c620 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800ea60 tx=0x7fd9d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.964+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800cde0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.964+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d80035f0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.964+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e81a2880 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.964+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8010690 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.964+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e81a8fe0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.965+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9e8063ad0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.968+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fd9d80037a0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.969+0000 7fd9df400640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9cc0b6670 0x7fd9cc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.969+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9d80e80a0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.969+0000 7fd9dd400640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9cc0b6670 0x7fd9cc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.970+0000 7fd9df400640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] conn(0x7fd9cc0c1c00 0x7fd9cc0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.970+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 --> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9d8011e50 con 0x7fd9cc0c1c00 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.970+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd9d80e84f0 con 0x7fd9e8105bb0 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.970+0000 7fd9dd400640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9cc0b6670 0x7fd9cc0b8b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd9d002aa00 tx=0x7fd9d0005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.970+0000 7fd9e6800640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] conn(0x7fd9cc0c1c00 0x7fd9cc0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.971+0000 7fd9e6800640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] conn(0x7fd9cc0c1c00 0x7fd9cc0c40f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.972+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== osd.1 v2:172.21.15.150:6800/4107270363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9d8011e50 con 0x7fd9cc0c1c00 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.986+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 --> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9e8063f50 con 0x7fd9cc0c1c00 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9df400640 1 -- 172.21.15.39:0/2204410944 <== osd.1 v2:172.21.15.150:6800/4107270363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd9e8063f50 con 0x7fd9cc0c1c00 2024-09-21T01:02:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] conn(0x7fd9cc0c1c00 msgr2=0x7fd9cc0c40f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] conn(0x7fd9cc0c1c00 0x7fd9cc0c40f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9cc0b6670 msgr2=0x7fd9cc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9cc0b6670 0x7fd9cc0b8b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd9d002aa00 tx=0x7fd9d0005a20 comp rx=0 tx=0).stop 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e8105bb0 msgr2=0x7fd9e819c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 --2- 172.21.15.39:0/2204410944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e8105bb0 0x7fd9e819c620 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800ea60 tx=0x7fd9d800ef20 comp rx=0 tx=0).stop 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e6800640 1 -- 172.21.15.39:0/2204410944 reap_dead start 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 shutdown_connections 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.987+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 >> 172.21.15.39:0/2204410944 conn(0x7fd9e8101680 msgr2=0x7fd9e810ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.988+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 shutdown_connections 2024-09-21T01:02:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:23.988+0000 7fd9e7200640 1 -- 172.21.15.39:0/2204410944 wait complete. 2024-09-21T01:02:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:27 smithi160 ceph-mon[34696]: pgmap v357: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:02:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:27 smithi039 ceph-mon[30415]: pgmap v357: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 -- 172.21.15.39:0/881298000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 msgr2=0x7f50000757b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 --2- 172.21.15.39:0/881298000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50000757b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009900 tx=0x7f4ff002a410 comp rx=0 tx=0).stop 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 -- 172.21.15.39:0/881298000 shutdown_connections 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 --2- 172.21.15.39:0/881298000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5000114450 0x7f5000116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 --2- 172.21.15.39:0/881298000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5000075dd0 0x7f50000761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 --2- 172.21.15.39:0/881298000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50000757b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:28.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 -- 172.21.15.39:0/881298000 >> 172.21.15.39:0/881298000 conn(0x7f50000ff590 msgr2=0x7f50001019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 -- 172.21.15.39:0/881298000 shutdown_connections 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.254+0000 7f5005e00640 1 -- 172.21.15.39:0/881298000 wait complete. 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.255+0000 7f5005e00640 1 Processor -- start 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.255+0000 7f5005e00640 1 -- start start 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.255+0000 7f5005e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5000075dd0 0x7f50001a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.255+0000 7f5005e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50001a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f5005e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5000114450 0x7f50001a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f5005e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f500011a140 con 0x7f5000077380 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f5005e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5000119fc0 con 0x7f5000075dd0 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f5005e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f500011a2c0 con 0x7f5000114450 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f4ffe000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50001a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f4ffe000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50001a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39996/0 (socket says 172.21.15.39:39996) 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f4ffe000640 1 -- 172.21.15.39:0/3773830977 learned_addr learned my addr 172.21.15.39:0/3773830977 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:28.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f4ffea00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5000075dd0 0x7f50001a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.256+0000 7f4fff400640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5000114450 0x7f50001a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffe000640 1 -- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5000114450 msgr2=0x7f50001a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffe000640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5000114450 0x7f50001a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffe000640 1 -- 172.21.15.39:0/3773830977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5000075dd0 msgr2=0x7f50001a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffe000640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5000075dd0 0x7f50001a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffe000640 1 -- 172.21.15.39:0/3773830977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50001a6a80 con 0x7f5000077380 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffea00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5000075dd0 0x7f50001a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f4ffe000640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50001a10b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00e9c0 tx=0x7f4fec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00cd80 con 0x7f5000077380 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50001a6d70 con 0x7f5000077380 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fec0054d0 con 0x7f5000077380 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.257+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec010430 con 0x7f5000077380 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.258+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50001ad550 con 0x7f5000077380 2024-09-21T01:02:28.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.259+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5000063b50 con 0x7f5000077380 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.261+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f4fec004f90 con 0x7f5000077380 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.264+0000 7f5004a00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4fe40b6670 0x7f4fe40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.264+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4fec0e8280 con 0x7f5000077380 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.264+0000 7f4ffea00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4fe40b6670 0x7f4fe40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.264+0000 7f5004a00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] conn(0x7f4fe40c1c00 0x7f4fe40c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.264+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 --> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fec015e80 con 0x7f4fe40c1c00 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.264+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f4fec0e8630 con 0x7f5000077380 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.265+0000 7f4ffea00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4fe40b6670 0x7f4fe40b8b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0000c00 tx=0x7f4ff0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.265+0000 7f4fff400640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] conn(0x7f4fe40c1c00 0x7f4fe40c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.266+0000 7f4fff400640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] conn(0x7f4fe40c1c00 0x7f4fe40c40f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.267+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== osd.8 v2:172.21.15.160:6816/4075583447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fec015e80 con 0x7f4fe40c1c00 2024-09-21T01:02:28.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.283+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 --> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5000063f50 con 0x7f4fe40c1c00 2024-09-21T01:02:28.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.283+0000 7f5004a00640 1 -- 172.21.15.39:0/3773830977 <== osd.8 v2:172.21.15.160:6816/4075583447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5000063f50 con 0x7f4fe40c1c00 2024-09-21T01:02:28.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] conn(0x7f4fe40c1c00 msgr2=0x7f4fe40c40f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:28.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] conn(0x7f4fe40c1c00 0x7f4fe40c40f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:28.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4fe40b6670 msgr2=0x7f4fe40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:28.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4fe40b6670 0x7f4fe40b8b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0000c00 tx=0x7f4ff0031400 comp rx=0 tx=0).stop 2024-09-21T01:02:28.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 msgr2=0x7f50001a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:28.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 --2- 172.21.15.39:0/3773830977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5000077380 0x7f50001a10b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00e9c0 tx=0x7f4fec00ee80 comp rx=0 tx=0).stop 2024-09-21T01:02:28.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f4fff400640 1 -- 172.21.15.39:0/3773830977 reap_dead start 2024-09-21T01:02:28.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 shutdown_connections 2024-09-21T01:02:28.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 >> 172.21.15.39:0/3773830977 conn(0x7f50000ff590 msgr2=0x7f5000110350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:28.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.284+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 shutdown_connections 2024-09-21T01:02:28.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:21.285+0000 7f5005e00640 1 -- 172.21.15.39:0/3773830977 wait complete. 2024-09-21T01:02:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:27 smithi150 ceph-mon[34945]: pgmap v357: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:02:28.737 INFO:teuthology.orchestra.run.smithi039.stdout:279172874278 2024-09-21T01:02:28.737 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.10 2024-09-21T01:02:29.073 INFO:teuthology.orchestra.run.smithi039.stdout:231928234035 2024-09-21T01:02:29.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.8 2024-09-21T01:02:29.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:29 smithi039 ceph-mon[30415]: pgmap v358: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-21T01:02:29.785 INFO:teuthology.orchestra.run.smithi039.stdout:60129542236 2024-09-21T01:02:29.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T01:02:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:29 smithi160 ceph-mon[34696]: pgmap v358: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-21T01:02:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:29 smithi150 ceph-mon[34945]: pgmap v358: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-21T01:02:30.079 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:30.079 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:30.119 INFO:teuthology.orchestra.run.smithi039.stdout:257698037806 2024-09-21T01:02:30.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.9 2024-09-21T01:02:30.846 INFO:teuthology.orchestra.run.smithi039.stdout:137438953547 2024-09-21T01:02:30.847 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T01:02:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 -- 172.21.15.39:0/435092870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c103b70 msgr2=0x7efe3c103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 --2- 172.21.15.39:0/435092870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c103b70 0x7efe3c103f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe30009860 tx=0x7efe3002a340 comp rx=0 tx=0).stop 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 -- 172.21.15.39:0/435092870 shutdown_connections 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 --2- 172.21.15.39:0/435092870 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe3c109e00 0x7efe3c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 --2- 172.21.15.39:0/435092870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe3c1094b0 0x7efe3c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 --2- 172.21.15.39:0/435092870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c103b70 0x7efe3c103f80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 -- 172.21.15.39:0/435092870 >> 172.21.15.39:0/435092870 conn(0x7efe3c0ff5e0 msgr2=0x7efe3c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 -- 172.21.15.39:0/435092870 shutdown_connections 2024-09-21T01:02:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.096+0000 7efe43800640 1 -- 172.21.15.39:0/435092870 wait complete. 2024-09-21T01:02:31.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.098+0000 7efe43800640 1 Processor -- start 2024-09-21T01:02:31.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.098+0000 7efe43800640 1 -- start start 2024-09-21T01:02:31.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.099+0000 7efe43800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe3c103b70 0x7efe3c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:31.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.099+0000 7efe43800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c1094b0 0x7efe3c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:31.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.099+0000 7efe43800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe3c109e00 0x7efe3c1a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.099+0000 7efe43800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c113ac0 con 0x7efe3c109e00 2024-09-21T01:02:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.099+0000 7efe43800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c113940 con 0x7efe3c1094b0 2024-09-21T01:02:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.099+0000 7efe43800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c113c40 con 0x7efe3c103b70 2024-09-21T01:02:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe41800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe3c109e00 0x7efe3c1a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe41800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe3c109e00 0x7efe3c1a1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43336/0 (socket says 172.21.15.39:43336) 2024-09-21T01:02:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe41800640 1 -- 172.21.15.39:0/367281249 learned_addr learned my addr 172.21.15.39:0/367281249 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:31.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe3be00640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c1094b0 0x7efe3c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:31.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe3be00640 1 -- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe3c103b70 msgr2=0x7efe3c19c690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:02:31.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe3be00640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efe3c103b70 0x7efe3c19c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe3be00640 1 -- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe3c109e00 msgr2=0x7efe3c1a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:31.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.100+0000 7efe3be00640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe3c109e00 0x7efe3c1a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.101+0000 7efe3be00640 1 -- 172.21.15.39:0/367281249 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c1a2600 con 0x7efe3c1094b0 2024-09-21T01:02:31.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.101+0000 7efe3be00640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c1094b0 0x7efe3c19cc30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efe3c06bd40 tx=0x7efe2800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:31.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.101+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2800cd00 con 0x7efe3c1094b0 2024-09-21T01:02:31.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.102+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe280035f0 con 0x7efe3c1094b0 2024-09-21T01:02:31.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.102+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28010690 con 0x7efe3c1094b0 2024-09-21T01:02:31.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.102+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3c1a28f0 con 0x7efe3c1094b0 2024-09-21T01:02:31.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.102+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe3c1a9050 con 0x7efe3c1094b0 2024-09-21T01:02:31.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.103+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efe3c110340 con 0x7efe3c1094b0 2024-09-21T01:02:31.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.108+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7efe28004020 con 0x7efe3c1094b0 2024-09-21T01:02:31.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.113+0000 7efe39600640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7efe240b66f0 0x7efe240b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:31.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.113+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7efe280e8050 con 0x7efe3c1094b0 2024-09-21T01:02:31.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.114+0000 7efe39600640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] conn(0x7efe240c1c80 0x7efe240c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:31.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.116+0000 7efe40e00640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7efe240b66f0 0x7efe240b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:31.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.118+0000 7efe41800640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] conn(0x7efe240c1c80 0x7efe240c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:31.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.119+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 --> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe28015e80 con 0x7efe240c1c80 2024-09-21T01:02:31.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.119+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7efe280e8440 con 0x7efe3c1094b0 2024-09-21T01:02:31.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.119+0000 7efe41800640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] conn(0x7efe240c1c80 0x7efe240c4170 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:31.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.119+0000 7efe40e00640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7efe240b66f0 0x7efe240b8c00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efe3002a860 tx=0x7efe30000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:31.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.121+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== osd.11 v2:172.21.15.160:6824/3176075747 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efe28015e80 con 0x7efe240c1c80 2024-09-21T01:02:31.135 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:31.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.151+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 --> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efe3c104df0 con 0x7efe240c1c80 2024-09-21T01:02:31.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.152+0000 7efe39600640 1 -- 172.21.15.39:0/367281249 <== osd.11 v2:172.21.15.160:6824/3176075747 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efe3c104df0 con 0x7efe240c1c80 2024-09-21T01:02:31.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.156+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] conn(0x7efe240c1c80 msgr2=0x7efe240c4170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:31.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.156+0000 7efe43800640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] conn(0x7efe240c1c80 0x7efe240c4170 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7efe240b66f0 msgr2=0x7efe240b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:31.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe43800640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7efe240b66f0 0x7efe240b8c00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efe3002a860 tx=0x7efe30000bc0 comp rx=0 tx=0).stop 2024-09-21T01:02:31.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c1094b0 msgr2=0x7efe3c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:31.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe43800640 1 --2- 172.21.15.39:0/367281249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe3c1094b0 0x7efe3c19cc30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efe3c06bd40 tx=0x7efe2800ee50 comp rx=0 tx=0).stop 2024-09-21T01:02:31.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe41800640 1 -- 172.21.15.39:0/367281249 reap_dead start 2024-09-21T01:02:31.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 shutdown_connections 2024-09-21T01:02:31.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.158+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 >> 172.21.15.39:0/367281249 conn(0x7efe3c0ff5e0 msgr2=0x7efe3c10e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:31.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.159+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 shutdown_connections 2024-09-21T01:02:31.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:31.159+0000 7efe43800640 1 -- 172.21.15.39:0/367281249 wait complete. 2024-09-21T01:02:31.162 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:31.171 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:31.172 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:31.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:31 smithi039 ceph-mon[30415]: pgmap v359: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-21T01:02:31.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:31 smithi160 ceph-mon[34696]: pgmap v359: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-21T01:02:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:31 smithi150 ceph-mon[34945]: pgmap v359: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-21T01:02:32.135 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:32.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.709+0000 7f0e7c400640 1 -- 172.21.15.39:0/3894303425 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e741049b0 msgr2=0x7f0e74104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.709+0000 7f0e7c400640 1 --2- 172.21.15.39:0/3894303425 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e741049b0 0x7f0e74104dc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009900 tx=0x7f0e6802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 -- 172.21.15.39:0/3894303425 shutdown_connections 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 --2- 172.21.15.39:0/3894303425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e7410de10 0x7f0e74110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 --2- 172.21.15.39:0/3894303425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74105300 0x7f0e7410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 --2- 172.21.15.39:0/3894303425 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e741049b0 0x7f0e74104dc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 -- 172.21.15.39:0/3894303425 >> 172.21.15.39:0/3894303425 conn(0x7f0e74079730 msgr2=0x7f0e74079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 -- 172.21.15.39:0/3894303425 shutdown_connections 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.710+0000 7f0e7c400640 1 -- 172.21.15.39:0/3894303425 wait complete. 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.711+0000 7f0e7c400640 1 Processor -- start 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.711+0000 7f0e7c400640 1 -- start start 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e7c400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e741049b0 0x7f0e7419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:32.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e7c400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e74105300 0x7f0e7419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e7c400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e7410de10 0x7f0e741a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e7c400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0e74113ac0 con 0x7f0e741049b0 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e7c400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e74113940 con 0x7f0e74105300 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e7c400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0e74113c40 con 0x7f0e7410de10 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e79a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e741049b0 0x7f0e7419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e79a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e741049b0 0x7f0e7419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43368/0 (socket says 172.21.15.39:43368) 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e79a00640 1 -- 172.21.15.39:0/1800163790 learned_addr learned my addr 172.21.15.39:0/1800163790 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.712+0000 7f0e79000640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e74105300 0x7f0e7419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79000640 1 -- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e7410de10 msgr2=0x7f0e741a1f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79000640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e7410de10 0x7f0e741a1f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79000640 1 -- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e741049b0 msgr2=0x7f0e7419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79000640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e741049b0 0x7f0e7419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79000640 1 -- 172.21.15.39:0/1800163790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e741a2640 con 0x7f0e74105300 2024-09-21T01:02:32.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79a00640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e741049b0 0x7f0e7419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e79000640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e74105300 0x7f0e7419cc70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0e7406ad30 tx=0x7f0e6000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6000cc40 con 0x7f0e74105300 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e6000cde0 con 0x7f0e74105300 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.713+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60018780 con 0x7f0e74105300 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.714+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e741a2930 con 0x7f0e74105300 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.714+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e741a9090 con 0x7f0e74105300 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.715+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e74104100 con 0x7f0e74105300 2024-09-21T01:02:32.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.717+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0e60010a90 con 0x7f0e74105300 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.719+0000 7f0e72000640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0e5c0b65f0 0x7f0e5c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.719+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0e600e84e0 con 0x7f0e74105300 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.719+0000 7f0e79a00640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0e5c0b65f0 0x7f0e5c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.719+0000 7f0e72000640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] conn(0x7f0e5c0c1b80 0x7f0e5c0c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.719+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 --> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e5c0c4740 con 0x7f0e5c0c1b80 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.719+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f0e600e8930 con 0x7f0e74105300 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.720+0000 7f0e79a00640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0e5c0b65f0 0x7f0e5c0b8b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e68000c00 tx=0x7f0e68005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.731+0000 7f0e7a400640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] conn(0x7f0e5c0c1b80 0x7f0e5c0c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.731+0000 7f0e7a400640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] conn(0x7f0e5c0c1b80 0x7f0e5c0c4070 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.732+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== osd.5 v2:172.21.15.160:6808/3500100397 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e5c0c4740 con 0x7f0e5c0c1b80 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.761+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 --> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e7406b560 con 0x7f0e5c0c1b80 2024-09-21T01:02:32.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e72000640 1 -- 172.21.15.39:0/1800163790 <== osd.5 v2:172.21.15.160:6808/3500100397 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0e7406b560 con 0x7f0e5c0c1b80 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] conn(0x7f0e5c0c1b80 msgr2=0x7f0e5c0c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] conn(0x7f0e5c0c1b80 0x7f0e5c0c4070 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0e5c0b65f0 msgr2=0x7f0e5c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0e5c0b65f0 0x7f0e5c0b8b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e68000c00 tx=0x7f0e68005a20 comp rx=0 tx=0).stop 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e74105300 msgr2=0x7f0e7419cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 --2- 172.21.15.39:0/1800163790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e74105300 0x7f0e7419cc70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0e7406ad30 tx=0x7f0e6000ee50 comp rx=0 tx=0).stop 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7a400640 1 -- 172.21.15.39:0/1800163790 reap_dead start 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 shutdown_connections 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 >> 172.21.15.39:0/1800163790 conn(0x7f0e74079730 msgr2=0x7f0e741036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 shutdown_connections 2024-09-21T01:02:32.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.762+0000 7f0e7c400640 1 -- 172.21.15.39:0/1800163790 wait complete. 2024-09-21T01:02:33.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 -- 172.21.15.39:0/2941171068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad4076100 msgr2=0x7f3ad410fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 --2- 172.21.15.39:0/2941171068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad4076100 0x7f3ad410fba0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009900 tx=0x7f3ac002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 -- 172.21.15.39:0/2941171068 shutdown_connections 2024-09-21T01:02:33.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 --2- 172.21.15.39:0/2941171068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad41100e0 0x7f3ad4112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 --2- 172.21.15.39:0/2941171068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad4076100 0x7f3ad410fba0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 --2- 172.21.15.39:0/2941171068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ad40757b0 0x7f3ad4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 -- 172.21.15.39:0/2941171068 >> 172.21.15.39:0/2941171068 conn(0x7f3ad40ff610 msgr2=0x7f3ad4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 -- 172.21.15.39:0/2941171068 shutdown_connections 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.816+0000 7f3adae00640 1 -- 172.21.15.39:0/2941171068 wait complete. 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.817+0000 7f3adae00640 1 Processor -- start 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.817+0000 7f3adae00640 1 -- start start 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3adae00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 0x7f3ad419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3adae00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ad4076100 0x7f3ad419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3adae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad41100e0 0x7f3ad41a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3adae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad4115c90 con 0x7f3ad41100e0 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3adae00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad4115b10 con 0x7f3ad40757b0 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3adae00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad4115e10 con 0x7f3ad4076100 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 0x7f3ad419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad41100e0 0x7f3ad41a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 0x7f3ad419c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:40564/0 (socket says 172.21.15.39:40564) 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 -- 172.21.15.39:0/984311766 learned_addr learned my addr 172.21.15.39:0/984311766 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3400640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ad4076100 0x7f3ad419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 -- 172.21.15.39:0/984311766 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ad4076100 msgr2=0x7f3ad419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ad4076100 0x7f3ad419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 -- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad41100e0 msgr2=0x7f3ad41a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.818+0000 7f3ad3e00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad41100e0 0x7f3ad41a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3ad3e00640 1 -- 172.21.15.39:0/984311766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad41a25f0 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3ad3e00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 0x7f3ad419c680 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400ccc0 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac40108e0 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4018880 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad41a28e0 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.819+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad41a9040 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.821+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad4076cc0 con 0x7f3ad40757b0 2024-09-21T01:02:33.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.823+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f3ac4010430 con 0x7f3ad40757b0 2024-09-21T01:02:33.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.826+0000 7f3ad0c00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ab80b65f0 0x7f3ab80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.826+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3ac40e82c0 con 0x7f3ad40757b0 2024-09-21T01:02:33.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.826+0000 7f3ad3400640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ab80b65f0 0x7f3ab80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.827+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3ac40ae4f0 con 0x7f3ad40757b0 2024-09-21T01:02:33.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:32.827+0000 7f3ad3400640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ab80b65f0 0x7f3ab80b8b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0000c00 tx=0x7f3ac0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:33.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.029+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3ad4075bc0 con 0x7f3ad40757b0 2024-09-21T01:02:33.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.030+0000 7f3ad0c00640 1 -- 172.21.15.39:0/984311766 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3ac40b28d0 con 0x7f3ad40757b0 2024-09-21T01:02:33.194 INFO:teuthology.orchestra.run.smithi039.stdout:184683593792 2024-09-21T01:02:33.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ab80b65f0 msgr2=0x7f3ab80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ab80b65f0 0x7f3ab80b8b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0000c00 tx=0x7f3ac0031400 comp rx=0 tx=0).stop 2024-09-21T01:02:33.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 msgr2=0x7f3ad419c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 0x7f3ad419c680 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 shutdown_connections 2024-09-21T01:02:33.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ad41100e0 0x7f3ad41a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3ad4076100 0x7f3ad419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3ab80b65f0 0x7f3ab80b8b00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 --2- 172.21.15.39:0/984311766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ad40757b0 0x7f3ad419c680 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 >> 172.21.15.39:0/984311766 conn(0x7f3ad40ff610 msgr2=0x7f3ad4109c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:33.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 shutdown_connections 2024-09-21T01:02:33.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.033+0000 7f3adae00640 1 -- 172.21.15.39:0/984311766 wait complete. 2024-09-21T01:02:33.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 -- 172.21.15.39:0/2984798617 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810b6c0 msgr2=0x7fde4810bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 --2- 172.21.15.39:0/2984798617 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810b6c0 0x7fde4810bad0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fde34009900 tx=0x7fde3402a400 comp rx=0 tx=0).stop 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 -- 172.21.15.39:0/2984798617 shutdown_connections 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 --2- 172.21.15.39:0/2984798617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810c010 0x7fde48112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 --2- 172.21.15.39:0/2984798617 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810b6c0 0x7fde4810bad0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 --2- 172.21.15.39:0/2984798617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d80 0x7fde48106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 -- 172.21.15.39:0/2984798617 >> 172.21.15.39:0/2984798617 conn(0x7fde48101810 msgr2=0x7fde48103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.830+0000 7fde50600640 1 -- 172.21.15.39:0/2984798617 shutdown_connections 2024-09-21T01:02:33.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.831+0000 7fde50600640 1 -- 172.21.15.39:0/2984798617 wait complete. 2024-09-21T01:02:33.832 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:33 smithi039 ceph-mon[30415]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-21T01:02:33.832 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:33 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/984311766' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T01:02:33.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 Processor -- start 2024-09-21T01:02:33.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 -- start start 2024-09-21T01:02:33.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde48105d80 0x7fde4819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 0x7fde4819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde4810c010 0x7fde481a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fde48115cd0 con 0x7fde4810b6c0 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde48115b50 con 0x7fde4810c010 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.832+0000 7fde50600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fde48115e50 con 0x7fde48105d80 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 0x7fde4819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4dc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde48105d80 0x7fde4819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 0x7fde4819cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34684/0 (socket says 172.21.15.39:34684) 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4d200640 1 -- 172.21.15.39:0/1279662804 learned_addr learned my addr 172.21.15.39:0/1279662804 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4e600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde4810c010 0x7fde481a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4dc00640 1 -- 172.21.15.39:0/1279662804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde4810c010 msgr2=0x7fde481a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4dc00640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde4810c010 0x7fde481a1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4dc00640 1 -- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 msgr2=0x7fde4819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4dc00640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 0x7fde4819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4dc00640 1 -- 172.21.15.39:0/1279662804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde481a2650 con 0x7fde48105d80 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.833+0000 7fde4d200640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 0x7fde4819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.834+0000 7fde4dc00640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde48105d80 0x7fde4819c6e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e9f0 tx=0x7fde3c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.834+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00ccc0 con 0x7fde48105d80 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.834+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde3c005020 con 0x7fde48105d80 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.834+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde481a2940 con 0x7fde48105d80 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.834+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00ce60 con 0x7fde48105d80 2024-09-21T01:02:33.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.834+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde481a90a0 con 0x7fde48105d80 2024-09-21T01:02:33.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.835+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde48106190 con 0x7fde48105d80 2024-09-21T01:02:33.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.838+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fde3c0051d0 con 0x7fde48105d80 2024-09-21T01:02:33.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.839+0000 7fde46000640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde300b6670 0x7fde300b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:33.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.839+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fde3c0e79c0 con 0x7fde48105d80 2024-09-21T01:02:33.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.839+0000 7fde4d200640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde300b6670 0x7fde300b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:33.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.840+0000 7fde4d200640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde300b6670 0x7fde300b8b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde4806b1e0 tx=0x7fde34032ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:33.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:33.841+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fde3c0adc80 con 0x7fde48105d80 2024-09-21T01:02:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:33 smithi160 ceph-mon[34696]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-21T01:02:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:33 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/984311766' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T01:02:34.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.044+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fde4819d810 con 0x7fde48105d80 2024-09-21T01:02:34.045 INFO:teuthology.orchestra.run.smithi039.stdout:81604378711 2024-09-21T01:02:34.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.045+0000 7fde46000640 1 -- 172.21.15.39:0/1279662804 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fde4819d810 con 0x7fde48105d80 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde300b6670 msgr2=0x7fde300b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde300b6670 0x7fde300b8b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde4806b1e0 tx=0x7fde34032ed0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde48105d80 msgr2=0x7fde4819c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde48105d80 0x7fde4819c6e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e9f0 tx=0x7fde3c00edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 shutdown_connections 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde300b6670 0x7fde300b8b80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde4810c010 0x7fde481a1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6c0 0x7fde4819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 --2- 172.21.15.39:0/1279662804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde48105d80 0x7fde4819c6e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.047+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 >> 172.21.15.39:0/1279662804 conn(0x7fde48101810 msgr2=0x7fde48103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:34.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.048+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 shutdown_connections 2024-09-21T01:02:34.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.048+0000 7fde50600640 1 -- 172.21.15.39:0/1279662804 wait complete. 2024-09-21T01:02:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:33 smithi150 ceph-mon[34945]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-21T01:02:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:33 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/984311766' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T01:02:34.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 -- 172.21.15.39:0/743061741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410c010 msgr2=0x7fd744112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 --2- 172.21.15.39:0/743061741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410c010 0x7fd744112840 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009ca0 tx=0x7fd73c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 -- 172.21.15.39:0/743061741 shutdown_connections 2024-09-21T01:02:34.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 --2- 172.21.15.39:0/743061741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410c010 0x7fd744112840 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 --2- 172.21.15.39:0/743061741 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd74410b6c0 0x7fd74410bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 --2- 172.21.15.39:0/743061741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd744105d80 0x7fd744106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 -- 172.21.15.39:0/743061741 >> 172.21.15.39:0/743061741 conn(0x7fd744101810 msgr2=0x7fd744103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:34.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 -- 172.21.15.39:0/743061741 shutdown_connections 2024-09-21T01:02:34.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.665+0000 7fd74c200640 1 -- 172.21.15.39:0/743061741 wait complete. 2024-09-21T01:02:34.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.666+0000 7fd74c200640 1 Processor -- start 2024-09-21T01:02:34.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.666+0000 7fd74c200640 1 -- start start 2024-09-21T01:02:34.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74c200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd744105d80 0x7fd74419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:34.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74c200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 0x7fd74419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74c200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd74410c010 0x7fd7441a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74c200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd744115d20 con 0x7fd74410b6c0 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74c200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd744115ba0 con 0x7fd74410c010 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74c200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd744115ea0 con 0x7fd744105d80 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd748e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 0x7fd74419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd748e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 0x7fd74419ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34696/0 (socket says 172.21.15.39:34696) 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd748e00640 1 -- 172.21.15.39:0/995362991 learned_addr learned my addr 172.21.15.39:0/995362991 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:34.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd74a200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd74410c010 0x7fd7441a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:34.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.667+0000 7fd749800640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd744105d80 0x7fd74419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:34.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74a200640 1 -- 172.21.15.39:0/995362991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd744105d80 msgr2=0x7fd74419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74a200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd744105d80 0x7fd74419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74a200640 1 -- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 msgr2=0x7fd74419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74a200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 0x7fd74419ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74a200640 1 -- 172.21.15.39:0/995362991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7441a2680 con 0x7fd74410c010 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd748e00640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 0x7fd74419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74a200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd74410c010 0x7fd7441a1f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd73c005e30 tx=0x7fd73c0317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c003b70 con 0x7fd74410c010 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd73c003d10 con 0x7fd74410c010 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c02f970 con 0x7fd74410c010 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.668+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7441a2910 con 0x7fd74410c010 2024-09-21T01:02:34.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.669+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7441a90d0 con 0x7fd74410c010 2024-09-21T01:02:34.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.670+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd744106190 con 0x7fd74410c010 2024-09-21T01:02:34.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.672+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fd73c004b40 con 0x7fd74410c010 2024-09-21T01:02:34.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.673+0000 7fd742000640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd72c0b65f0 0x7fd72c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:34.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.674+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd73c108b60 con 0x7fd74410c010 2024-09-21T01:02:34.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.674+0000 7fd749800640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd72c0b65f0 0x7fd72c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:34.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.674+0000 7fd749800640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd72c0b65f0 0x7fd72c0b8b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd738009bd0 tx=0x7fd738005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:34.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.675+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd73c0ced90 con 0x7fd74410c010 2024-09-21T01:02:34.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:34 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1279662804' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T01:02:34.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.873+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd74402e640 con 0x7fd74410c010 2024-09-21T01:02:34.874 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:34 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1279662804' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T01:02:34.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.874+0000 7fd742000640 1 -- 172.21.15.39:0/995362991 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd73c0d3170 con 0x7fd74410c010 2024-09-21T01:02:34.874 INFO:teuthology.orchestra.run.smithi039.stdout:38654705763 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd72c0b65f0 msgr2=0x7fd72c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd72c0b65f0 0x7fd72c0b8b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd738009bd0 tx=0x7fd738005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd74410c010 msgr2=0x7fd7441a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd74410c010 0x7fd7441a1f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd73c005e30 tx=0x7fd73c0317c0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 shutdown_connections 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd72c0b65f0 0x7fd72c0b8b00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd74410c010 0x7fd7441a1f50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd74410b6c0 0x7fd74419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 --2- 172.21.15.39:0/995362991 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd744105d80 0x7fd74419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:34.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.876+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 >> 172.21.15.39:0/995362991 conn(0x7fd744101810 msgr2=0x7fd74410acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:34.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.877+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 shutdown_connections 2024-09-21T01:02:34.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:34.877+0000 7fd74c200640 1 -- 172.21.15.39:0/995362991 wait complete. 2024-09-21T01:02:35.036 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593790 got 184683593792 for osd.6 2024-09-21T01:02:35.037 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:35.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:34 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1279662804' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T01:02:35.634 INFO:teuthology.orchestra.run.smithi039.stdout:304942678052 2024-09-21T01:02:35.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.11 2024-09-21T01:02:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:35.816 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:35 smithi039 ceph-mon[30415]: pgmap v361: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-21T01:02:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:35 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/995362991' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T01:02:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:35 smithi150 ceph-mon[34945]: pgmap v361: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-21T01:02:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:35 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/995362991' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T01:02:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:35 smithi160 ceph-mon[34696]: pgmap v361: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-21T01:02:36.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:35 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/995362991' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T01:02:36.346 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378710 got 81604378711 for osd.2 2024-09-21T01:02:36.347 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:36.445 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 -- 172.21.15.39:0/1713343758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c0109c20 msgr2=0x7f41c0110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 --2- 172.21.15.39:0/1713343758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c0109c20 0x7f41c0110450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009be0 tx=0x7f41b802a6c0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 -- 172.21.15.39:0/1713343758 shutdown_connections 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 --2- 172.21.15.39:0/1713343758 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c0109c20 0x7f41c0110450 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 --2- 172.21.15.39:0/1713343758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c01092d0 0x7f41c01096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 --2- 172.21.15.39:0/1713343758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0103990 0x7f41c0103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 -- 172.21.15.39:0/1713343758 >> 172.21.15.39:0/1713343758 conn(0x7f41c00ff440 msgr2=0x7f41c0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 -- 172.21.15.39:0/1713343758 shutdown_connections 2024-09-21T01:02:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.623+0000 7f41bfe00640 1 -- 172.21.15.39:0/1713343758 wait complete. 2024-09-21T01:02:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.624+0000 7f41bfe00640 1 Processor -- start 2024-09-21T01:02:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.624+0000 7f41bfe00640 1 -- start start 2024-09-21T01:02:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c0103990 0x7f41c019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c01092d0 0x7f41c019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bfe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0109c20 0x7f41c01a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01138e0 con 0x7f41c0103990 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bfe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0113760 con 0x7f41c0109c20 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bfe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0113a60 con 0x7f41c01092d0 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c0103990 0x7f41c019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41b7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c01092d0 0x7f41c019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c0103990 0x7f41c019c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34714/0 (socket says 172.21.15.39:34714) 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bea00640 1 -- 172.21.15.39:0/4257988699 learned_addr learned my addr 172.21.15.39:0/4257988699 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41bf400640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0109c20 0x7f41c01a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.625+0000 7f41b7e00640 1 -- 172.21.15.39:0/4257988699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0109c20 msgr2=0x7f41c01a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41b7e00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0109c20 0x7f41c01a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41b7e00640 1 -- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c0103990 msgr2=0x7f41c019c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41b7e00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c0103990 0x7f41c019c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41b7e00640 1 -- 172.21.15.39:0/4257988699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c01a2440 con 0x7f41c01092d0 2024-09-21T01:02:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41bf400640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0109c20 0x7f41c01a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41b7e00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c01092d0 0x7f41c019ca70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f41c006ca20 tx=0x7f41a800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a800cc90 con 0x7f41c01092d0 2024-09-21T01:02:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41a800ee70 con 0x7f41c01092d0 2024-09-21T01:02:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8018660 con 0x7f41c01092d0 2024-09-21T01:02:36.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c01a2730 con 0x7f41c01092d0 2024-09-21T01:02:36.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.626+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c01a8e90 con 0x7f41c01092d0 2024-09-21T01:02:36.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.628+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c0103da0 con 0x7f41c01092d0 2024-09-21T01:02:36.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.630+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f41a8010430 con 0x7f41c01092d0 2024-09-21T01:02:36.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.631+0000 7f41c4a00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41a40b65f0 0x7f41a40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:36.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.631+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f41a80e7c40 con 0x7f41c01092d0 2024-09-21T01:02:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.631+0000 7f41bea00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41a40b65f0 0x7f41a40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.632+0000 7f41bea00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41a40b65f0 0x7f41a40b8b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009c90 tx=0x7f41b0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:36.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.635+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f41a80ade70 con 0x7f41c01092d0 2024-09-21T01:02:36.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.844+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f41c01a9060 con 0x7f41c01092d0 2024-09-21T01:02:36.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.844+0000 7f41c4a00640 1 -- 172.21.15.39:0/4257988699 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f41a80b2250 con 0x7f41c01092d0 2024-09-21T01:02:36.845 INFO:teuthology.orchestra.run.smithi039.stdout:206158430267 2024-09-21T01:02:36.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41a40b65f0 msgr2=0x7f41a40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:36.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41a40b65f0 0x7f41a40b8b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009c90 tx=0x7f41b0005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:36.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c01092d0 msgr2=0x7f41c019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c01092d0 0x7f41c019ca70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f41c006ca20 tx=0x7f41a800ec20 comp rx=0 tx=0).stop 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 shutdown_connections 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41a40b65f0 0x7f41a40b8b00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c0109c20 0x7f41c01a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41c01092d0 0x7f41c019ca70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 --2- 172.21.15.39:0/4257988699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41c0103990 0x7f41c019c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 >> 172.21.15.39:0/4257988699 conn(0x7f41c00ff440 msgr2=0x7f41c0106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 shutdown_connections 2024-09-21T01:02:36.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:36.847+0000 7f41bfe00640 1 -- 172.21.15.39:0/4257988699 wait complete. 2024-09-21T01:02:37.895 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:37 smithi039 ceph-mon[30415]: pgmap v362: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:37.895 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:37 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4257988699' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T01:02:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 -- 172.21.15.39:0/1009827471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c105bb0 msgr2=0x7f4f6c105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 --2- 172.21.15.39:0/1009827471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c105fc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009900 tx=0x7f4f6002a410 comp rx=0 tx=0).stop 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 -- 172.21.15.39:0/1009827471 shutdown_connections 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 --2- 172.21.15.39:0/1009827471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f6c10be40 0x7f4f6c112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 --2- 172.21.15.39:0/1009827471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 --2- 172.21.15.39:0/1009827471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c105fc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 -- 172.21.15.39:0/1009827471 >> 172.21.15.39:0/1009827471 conn(0x7f4f6c101680 msgr2=0x7f4f6c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.948+0000 7f4f72600640 1 -- 172.21.15.39:0/1009827471 shutdown_connections 2024-09-21T01:02:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.949+0000 7f4f72600640 1 -- 172.21.15.39:0/1009827471 wait complete. 2024-09-21T01:02:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.950+0000 7f4f72600640 1 Processor -- start 2024-09-21T01:02:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 -- start start 2024-09-21T01:02:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f6c10be40 0x7f4f6c1a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c115b00 con 0x7f4f6c10b4f0 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c115980 con 0x7f4f6c10be40 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f72600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c115c80 con 0x7f4f6c105bb0 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.951+0000 7f4f6be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f6be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c19cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34744/0 (socket says 172.21.15.39:34744) 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f6be00640 1 -- 172.21.15.39:0/16913728 learned_addr learned my addr 172.21.15.39:0/16913728 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:37.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71c00640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f6c10be40 0x7f4f6c1a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:37.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 -- 172.21.15.39:0/16913728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f6c10be40 msgr2=0x7f4f6c1a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:37.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f6c10be40 0x7f4f6c1a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:37.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 -- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 msgr2=0x7f4f6c19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:37.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 -- 172.21.15.39:0/16913728 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c1a2480 con 0x7f4f6c105bb0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f6be00640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.952+0000 7f4f71200640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c19c510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f6002a930 tx=0x7f4f60031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.953+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6003f030 con 0x7f4f6c105bb0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.953+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60004490 con 0x7f4f6c105bb0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.953+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60035070 con 0x7f4f6c105bb0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.953+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6c1a2710 con 0x7f4f6c105bb0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.953+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f6c1a8ed0 con 0x7f4f6c105bb0 2024-09-21T01:02:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.954+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f6c106e30 con 0x7f4f6c105bb0 2024-09-21T01:02:37.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.957+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f4f60042030 con 0x7f4f6c105bb0 2024-09-21T01:02:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.958+0000 7f4f69600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f540b65f0 0x7f4f540b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.958+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4f60107eb0 con 0x7f4f6c105bb0 2024-09-21T01:02:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.958+0000 7f4f6be00640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f540b65f0 0x7f4f540b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:37.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.959+0000 7f4f6be00640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f540b65f0 0x7f4f540b8b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c10b000 tx=0x7f4f58005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:37.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:37.960+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4f600d24c0 con 0x7f4f6c105bb0 2024-09-21T01:02:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:37 smithi150 ceph-mon[34945]: pgmap v362: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:37 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4257988699' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T01:02:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:37 smithi160 ceph-mon[34696]: pgmap v362: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:37 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4257988699' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T01:02:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.166+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f4f6c19d640 con 0x7f4f6c105bb0 2024-09-21T01:02:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.167+0000 7f4f69600640 1 -- 172.21.15.39:0/16913728 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4f600d7400 con 0x7f4f6c105bb0 2024-09-21T01:02:38.168 INFO:teuthology.orchestra.run.smithi039.stdout:231928234038 2024-09-21T01:02:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f540b65f0 msgr2=0x7f4f540b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f540b65f0 0x7f4f540b8b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c10b000 tx=0x7f4f58005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c105bb0 msgr2=0x7f4f6c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c19c510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f6002a930 tx=0x7f4f60031a20 comp rx=0 tx=0).stop 2024-09-21T01:02:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 shutdown_connections 2024-09-21T01:02:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f540b65f0 0x7f4f540b8b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f6c10be40 0x7f4f6c1a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6c10b4f0 0x7f4f6c19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 --2- 172.21.15.39:0/16913728 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4f6c105bb0 0x7f4f6c19c510 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.170+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 >> 172.21.15.39:0/16913728 conn(0x7f4f6c101680 msgr2=0x7f4f6c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.171+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 shutdown_connections 2024-09-21T01:02:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.171+0000 7f4f72600640 1 -- 172.21.15.39:0/16913728 wait complete. 2024-09-21T01:02:38.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 -- 172.21.15.39:0/2939600632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba90105bc0 msgr2=0x7fba90105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 --2- 172.21.15.39:0/2939600632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba90105bc0 0x7fba90105fd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba80009860 tx=0x7fba8002a340 comp rx=0 tx=0).stop 2024-09-21T01:02:38.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 -- 172.21.15.39:0/2939600632 shutdown_connections 2024-09-21T01:02:38.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 --2- 172.21.15.39:0/2939600632 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba9010be50 0x7fba90112680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 --2- 172.21.15.39:0/2939600632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9010b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 --2- 172.21.15.39:0/2939600632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba90105bc0 0x7fba90105fd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 -- 172.21.15.39:0/2939600632 >> 172.21.15.39:0/2939600632 conn(0x7fba90101670 msgr2=0x7fba90103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.218+0000 7fba8f200640 1 -- 172.21.15.39:0/2939600632 shutdown_connections 2024-09-21T01:02:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.219+0000 7fba8f200640 1 -- 172.21.15.39:0/2939600632 wait complete. 2024-09-21T01:02:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 Processor -- start 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 -- start start 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba90105bc0 0x7fba9019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9010be50 0x7fba901a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fba90115b10 con 0x7fba9010b500 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba90115990 con 0x7fba9010be50 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.220+0000 7fba8f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fba90115c90 con 0x7fba90105bc0 2024-09-21T01:02:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34764/0 (socket says 172.21.15.39:34764) 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 -- 172.21.15.39:0/2958901856 learned_addr learned my addr 172.21.15.39:0/2958901856 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8de00640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba90105bc0 0x7fba9019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8e800640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9010be50 0x7fba901a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 -- 172.21.15.39:0/2958901856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba90105bc0 msgr2=0x7fba9019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba90105bc0 0x7fba9019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 -- 172.21.15.39:0/2958901856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9010be50 msgr2=0x7fba901a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9010be50 0x7fba901a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8d400640 1 -- 172.21.15.39:0/2958901856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba901a2470 con 0x7fba9010b500 2024-09-21T01:02:38.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8de00640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba90105bc0 0x7fba9019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:38.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.221+0000 7fba8e800640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9010be50 0x7fba901a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.222+0000 7fba8d400640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9019caa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00e9f0 tx=0x7fba7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.222+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c00ccc0 con 0x7fba9010b500 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.222+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba7c0108e0 con 0x7fba9010b500 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.222+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba901a2760 con 0x7fba9010b500 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.222+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c018860 con 0x7fba9010b500 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.222+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba901a8ec0 con 0x7fba9010b500 2024-09-21T01:02:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.223+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba90105fd0 con 0x7fba9010b500 2024-09-21T01:02:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.226+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fba7c010430 con 0x7fba9010b500 2024-09-21T01:02:38.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.229+0000 7fba86a00640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba740b6670 0x7fba740b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:38.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.229+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fba7c0e8200 con 0x7fba9010b500 2024-09-21T01:02:38.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.229+0000 7fba8de00640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba740b6670 0x7fba740b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:38.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.230+0000 7fba8de00640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba740b6670 0x7fba740b8b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fba80009830 tx=0x7fba80030340 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.233+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fba7c0ae430 con 0x7fba9010b500 2024-09-21T01:02:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.444+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fba9019d630 con 0x7fba9010b500 2024-09-21T01:02:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.444+0000 7fba86a00640 1 -- 172.21.15.39:0/2958901856 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fba9019d630 con 0x7fba9010b500 2024-09-21T01:02:38.445 INFO:teuthology.orchestra.run.smithi039.stdout:111669149778 2024-09-21T01:02:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba740b6670 msgr2=0x7fba740b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba740b6670 0x7fba740b8b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fba80009830 tx=0x7fba80030340 comp rx=0 tx=0).stop 2024-09-21T01:02:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 msgr2=0x7fba9019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9019caa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00e9f0 tx=0x7fba7c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 shutdown_connections 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba740b6670 0x7fba740b8b80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9010be50 0x7fba901a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba9010b500 0x7fba9019caa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 --2- 172.21.15.39:0/2958901856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba90105bc0 0x7fba9019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.447+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 >> 172.21.15.39:0/2958901856 conn(0x7fba90101670 msgr2=0x7fba9010abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.448+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 shutdown_connections 2024-09-21T01:02:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.448+0000 7fba8f200640 1 -- 172.21.15.39:0/2958901856 wait complete. 2024-09-21T01:02:38.816 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:38 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/16913728' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T01:02:38.816 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:38 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2958901856' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T01:02:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 -- 172.21.15.39:0/4253159822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab8109c40 msgr2=0x7f0ab8110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/4253159822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab8109c40 0x7f0ab8110470 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009900 tx=0x7f0ab002a410 comp rx=0 tx=0).stop 2024-09-21T01:02:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 -- 172.21.15.39:0/4253159822 shutdown_connections 2024-09-21T01:02:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/4253159822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab8109c40 0x7f0ab8110470 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/4253159822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ab81092f0 0x7f0ab8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/4253159822 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0ab81039b0 0x7f0ab8103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 -- 172.21.15.39:0/4253159822 >> 172.21.15.39:0/4253159822 conn(0x7f0ab80ff440 msgr2=0x7f0ab8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 -- 172.21.15.39:0/4253159822 shutdown_connections 2024-09-21T01:02:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.997+0000 7f0ab7e00640 1 -- 172.21.15.39:0/4253159822 wait complete. 2024-09-21T01:02:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.998+0000 7f0ab7e00640 1 Processor -- start 2024-09-21T01:02:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 -- start start 2024-09-21T01:02:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 0x7f0ab819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0ab81092f0 0x7f0ab819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ab8109c40 0x7f0ab819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab8113950 con 0x7f0ab81039b0 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81137d0 con 0x7f0ab8109c40 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab8113ad0 con 0x7f0ab81092f0 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 0x7f0ab819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 0x7f0ab819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34784/0 (socket says 172.21.15.39:34784) 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:38.999+0000 7f0ab6a00640 1 -- 172.21.15.39:0/773186277 learned_addr learned my addr 172.21.15.39:0/773186277 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab7400640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ab8109c40 0x7f0ab819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:39.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0aae000640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0ab81092f0 0x7f0ab819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab6a00640 1 -- 172.21.15.39:0/773186277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0ab81092f0 msgr2=0x7f0ab819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab6a00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0ab81092f0 0x7f0ab819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab6a00640 1 -- 172.21.15.39:0/773186277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ab8109c40 msgr2=0x7f0ab819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab6a00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ab8109c40 0x7f0ab819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab6a00640 1 -- 172.21.15.39:0/773186277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab81a0250 con 0x7f0ab81039b0 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.000+0000 7f0ab6a00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 0x7f0ab819a2e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800eb90 tx=0x7f0aa800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.001+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8004d20 con 0x7f0ab81039b0 2024-09-21T01:02:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.001+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0aa800ee90 con 0x7f0ab81039b0 2024-09-21T01:02:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.001+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab81a0540 con 0x7f0ab81039b0 2024-09-21T01:02:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.001+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8010670 con 0x7f0ab81039b0 2024-09-21T01:02:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.001+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab81a6ca0 con 0x7f0ab81039b0 2024-09-21T01:02:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.002+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab8103dc0 con 0x7f0ab81039b0 2024-09-21T01:02:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.005+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0aa8005020 con 0x7f0ab81039b0 2024-09-21T01:02:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.007+0000 7f0aafe00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a9c0b6670 0x7f0a9c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.007+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0aa80e7fd0 con 0x7f0ab81039b0 2024-09-21T01:02:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.007+0000 7f0aae000640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a9c0b6670 0x7f0a9c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.008+0000 7f0aae000640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a9c0b6670 0x7f0a9c0b8b80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8102c90 tx=0x7f0aa0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:39.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.011+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0aa80ae200 con 0x7f0ab81039b0 2024-09-21T01:02:39.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:38 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/16913728' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T01:02:39.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:38 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2958901856' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T01:02:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:38 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/16913728' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T01:02:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:38 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2958901856' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T01:02:39.137 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705760 got 38654705763 for osd.0 2024-09-21T01:02:39.137 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:39.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.225+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0ab810b4a0 con 0x7f0ab81039b0 2024-09-21T01:02:39.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.225+0000 7f0aafe00640 1 -- 172.21.15.39:0/773186277 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0aa80b25e0 con 0x7f0ab81039b0 2024-09-21T01:02:39.226 INFO:teuthology.orchestra.run.smithi039.stdout:60129542239 2024-09-21T01:02:39.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a9c0b6670 msgr2=0x7f0a9c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:39.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a9c0b6670 0x7f0a9c0b8b80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8102c90 tx=0x7f0aa0005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 msgr2=0x7f0ab819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 0x7f0ab819a2e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800eb90 tx=0x7f0aa800c6a0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 shutdown_connections 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a9c0b6670 0x7f0a9c0b8b80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ab8109c40 0x7f0ab819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0ab81092f0 0x7f0ab819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 --2- 172.21.15.39:0/773186277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab81039b0 0x7f0ab819a2e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 >> 172.21.15.39:0/773186277 conn(0x7f0ab80ff440 msgr2=0x7f0ab8108930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 shutdown_connections 2024-09-21T01:02:39.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:39.228+0000 7f0ab7e00640 1 -- 172.21.15.39:0/773186277 wait complete. 2024-09-21T01:02:39.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:39 smithi039 ceph-mon[30415]: pgmap v363: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:39.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:39 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/773186277' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T01:02:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:39 smithi150 ceph-mon[34945]: pgmap v363: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:39 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/773186277' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T01:02:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:39 smithi160 ceph-mon[34696]: pgmap v363: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:39 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/773186277' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T01:02:40.153 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430264 got 206158430267 for osd.7 2024-09-21T01:02:40.154 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:40.500 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149775 got 111669149778 for osd.3 2024-09-21T01:02:40.500 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:41.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 -- 172.21.15.39:0/1909872715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 msgr2=0x7f304c112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:41.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 --2- 172.21.15.39:0/1909872715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c112840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3044009ca0 tx=0x7f304402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 -- 172.21.15.39:0/1909872715 shutdown_connections 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 --2- 172.21.15.39:0/1909872715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c112840 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 --2- 172.21.15.39:0/1909872715 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 0x7f304c10bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 --2- 172.21.15.39:0/1909872715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 -- 172.21.15.39:0/1909872715 >> 172.21.15.39:0/1909872715 conn(0x7f304c1017d0 msgr2=0x7f304c103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 -- 172.21.15.39:0/1909872715 shutdown_connections 2024-09-21T01:02:41.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.208+0000 7f3052400640 1 -- 172.21.15.39:0/1909872715 wait complete. 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.209+0000 7f3052400640 1 Processor -- start 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.209+0000 7f3052400640 1 -- start start 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3052400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3052400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 0x7f304c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3052400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c1a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3052400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f304c115cd0 con 0x7f304c10c010 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3052400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f304c115b50 con 0x7f304c105d80 2024-09-21T01:02:41.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3052400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f304c115e50 con 0x7f304c10b6c0 2024-09-21T01:02:41.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f304b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f304be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c1a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f304b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:56136/0 (socket says 172.21.15.39:56136) 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f304b400640 1 -- 172.21.15.39:0/3512093410 learned_addr learned my addr 172.21.15.39:0/3512093410 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.210+0000 7f3042a00640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 0x7f304c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304b400640 1 -- 172.21.15.39:0/3512093410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 msgr2=0x7f304c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304b400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 0x7f304c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304b400640 1 -- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 msgr2=0x7f304c1a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304b400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c1a1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304b400640 1 -- 172.21.15.39:0/3512093410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304c1a25d0 con 0x7f304c105d80 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f3042a00640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 0x7f304c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304be00640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c1a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f304b400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c19c660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f303c00e9f0 tx=0x7f303c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c00cd80 con 0x7f304c105d80 2024-09-21T01:02:41.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f303c0108e0 con 0x7f304c105d80 2024-09-21T01:02:41.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304c1a28c0 con 0x7f304c105d80 2024-09-21T01:02:41.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.211+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c018980 con 0x7f304c105d80 2024-09-21T01:02:41.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.212+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304c1a9020 con 0x7f304c105d80 2024-09-21T01:02:41.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.213+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f304c106190 con 0x7f304c105d80 2024-09-21T01:02:41.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.215+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f303c010430 con 0x7f304c105d80 2024-09-21T01:02:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.217+0000 7f3048c00640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f30300b65f0 0x7f30300b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.217+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f303c0e81b0 con 0x7f304c105d80 2024-09-21T01:02:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.217+0000 7f3042a00640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f30300b65f0 0x7f30300b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.217+0000 7f3042a00640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f30300b65f0 0x7f30300b8b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f30340095f0 tx=0x7f3034005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.218+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f303c0b27c0 con 0x7f304c105d80 2024-09-21T01:02:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.438+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f304c02e640 con 0x7f304c105d80 2024-09-21T01:02:41.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.439+0000 7f3048c00640 1 -- 172.21.15.39:0/3512093410 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f303c0b7700 con 0x7f304c105d80 2024-09-21T01:02:41.440 INFO:teuthology.orchestra.run.smithi039.stdout:137438953550 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f30300b65f0 msgr2=0x7f30300b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f30300b65f0 0x7f30300b8b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f30340095f0 tx=0x7f3034005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 msgr2=0x7f304c19c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c19c660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f303c00e9f0 tx=0x7f303c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 shutdown_connections 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f304c10c010 0x7f304c1a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f304c10b6c0 0x7f304c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f30300b65f0 0x7f30300b8b00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 --2- 172.21.15.39:0/3512093410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f304c105d80 0x7f304c19c660 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 >> 172.21.15.39:0/3512093410 conn(0x7f304c1017d0 msgr2=0x7f304c103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 shutdown_connections 2024-09-21T01:02:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:41.441+0000 7f3052400640 1 -- 172.21.15.39:0/3512093410 wait complete. 2024-09-21T01:02:41.741 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234035 got 231928234038 for osd.8 2024-09-21T01:02:41.741 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: pgmap v364: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3512093410' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: pgmap v364: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3512093410' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: pgmap v364: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3512093410' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:42.395 INFO:teuthology.orchestra.run.smithi039.stdout:158913790023 2024-09-21T01:02:42.395 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 -- 172.21.15.39:0/2302475519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 msgr2=0x7ff4a0106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2302475519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a0106190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff490009900 tx=0x7ff49002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 -- 172.21.15.39:0/2302475519 shutdown_connections 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2302475519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a010c010 0x7ff4a0112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2302475519 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 0x7ff4a010bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2302475519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a0106190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 -- 172.21.15.39:0/2302475519 >> 172.21.15.39:0/2302475519 conn(0x7ff4a0101810 msgr2=0x7ff4a0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 -- 172.21.15.39:0/2302475519 shutdown_connections 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.945+0000 7ff4a6800640 1 -- 172.21.15.39:0/2302475519 wait complete. 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 Processor -- start 2024-09-21T01:02:42.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 -- start start 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 0x7ff4a019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a010c010 0x7ff4a01a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a0115d20 con 0x7ff4a010c010 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a0115ba0 con 0x7ff4a0105d80 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.946+0000 7ff4a6800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a0115ea0 con 0x7ff4a010b6c0 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a019c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:33414/0 (socket says 172.21.15.39:33414) 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff497e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 0x7ff4a019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 -- 172.21.15.39:0/2246464438 learned_addr learned my addr 172.21.15.39:0/2246464438 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49fe00640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a010c010 0x7ff4a01a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 -- 172.21.15.39:0/2246464438 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 msgr2=0x7ff4a019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 0x7ff4a019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:42.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 -- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a010c010 msgr2=0x7ff4a01a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a010c010 0x7ff4a01a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 -- 172.21.15.39:0/2246464438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a01a2600 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff497e00640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 0x7ff4a019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.947+0000 7ff49f400640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a019c690 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff490005ab0 tx=0x7ff490003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.948+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49003f070 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.948+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff490031b60 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.948+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a01a2890 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.948+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490035030 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.948+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a01a9050 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.949+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a0106190 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.951+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7ff490042070 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.953+0000 7ff49cc00640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4840b65f0 0x7ff4840b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.953+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff490107ff0 con 0x7ff4a0105d80 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.953+0000 7ff497e00640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4840b65f0 0x7ff4840b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.954+0000 7ff497e00640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4840b65f0 0x7ff4840b8b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0105060 tx=0x7ff488005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:42.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:42.955+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff4900d2600 con 0x7ff4a0105d80 2024-09-21T01:02:43.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:43.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:42 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:43.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:42 smithi150 ceph-mon[34945]: pgmap v365: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:02:43.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:43.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:42 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:43.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:42 smithi160 ceph-mon[34696]: pgmap v365: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:02:43.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:43.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:42 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:43.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:42 smithi039 ceph-mon[30415]: pgmap v365: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:02:43.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.162+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff4a010dde0 con 0x7ff4a0105d80 2024-09-21T01:02:43.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.162+0000 7ff49cc00640 1 -- 172.21.15.39:0/2246464438 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff4900d7540 con 0x7ff4a0105d80 2024-09-21T01:02:43.164 INFO:teuthology.orchestra.run.smithi039.stdout:279172874282 2024-09-21T01:02:43.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4840b65f0 msgr2=0x7ff4840b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:43.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4840b65f0 0x7ff4840b8b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0105060 tx=0x7ff488005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:43.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 msgr2=0x7ff4a019c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:43.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a019c690 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff490005ab0 tx=0x7ff490003c10 comp rx=0 tx=0).stop 2024-09-21T01:02:43.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 shutdown_connections 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a010c010 0x7ff4a01a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a010b6c0 0x7ff4a019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff4840b65f0 0x7ff4840b8b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 --2- 172.21.15.39:0/2246464438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a0105d80 0x7ff4a019c690 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 >> 172.21.15.39:0/2246464438 conn(0x7ff4a0101810 msgr2=0x7ff4a010ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 shutdown_connections 2024-09-21T01:02:43.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:43.165+0000 7ff4a6800640 1 -- 172.21.15.39:0/2246464438 wait complete. 2024-09-21T01:02:43.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953547 got 137438953550 for osd.4 2024-09-21T01:02:43.354 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:44.240 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.240 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.240 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.240 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:43 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.240 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:43 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2246464438' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 -- 172.21.15.39:0/876089420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 msgr2=0x7f281810b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 --2- 172.21.15.39:0/876089420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281810b920 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2800009900 tx=0x7f280002a410 comp rx=0 tx=0).stop 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 -- 172.21.15.39:0/876089420 shutdown_connections 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 --2- 172.21.15.39:0/876089420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281810be60 0x7f2818112690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 --2- 172.21.15.39:0/876089420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281810b920 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 --2- 172.21.15.39:0/876089420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2818105bd0 0x7f2818105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 -- 172.21.15.39:0/876089420 >> 172.21.15.39:0/876089420 conn(0x7f2818101620 msgr2=0x7f2818103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 -- 172.21.15.39:0/876089420 shutdown_connections 2024-09-21T01:02:44.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.304+0000 7f281cc00640 1 -- 172.21.15.39:0/876089420 wait complete. 2024-09-21T01:02:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.306+0000 7f281cc00640 1 Processor -- start 2024-09-21T01:02:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.306+0000 7f281cc00640 1 -- start start 2024-09-21T01:02:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f281cc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2818105bd0 0x7f281819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f281cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f281cc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281810be60 0x7f28181a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f281cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2818115b20 con 0x7f281810b510 2024-09-21T01:02:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f281cc00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28181159a0 con 0x7f281810be60 2024-09-21T01:02:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f281cc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2818115ca0 con 0x7f2818105bd0 2024-09-21T01:02:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f280fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f280fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33524/0 (socket says 172.21.15.39:33524) 2024-09-21T01:02:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f280fe00640 1 -- 172.21.15.39:0/2441001616 learned_addr learned my addr 172.21.15.39:0/2441001616 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f2817400640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2818105bd0 0x7f281819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f2817e00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281810be60 0x7f28181a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f280fe00640 1 -- 172.21.15.39:0/2441001616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2818105bd0 msgr2=0x7f281819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.307+0000 7f280fe00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2818105bd0 0x7f281819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f280fe00640 1 -- 172.21.15.39:0/2441001616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281810be60 msgr2=0x7f28181a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f280fe00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281810be60 0x7f28181a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f280fe00640 1 -- 172.21.15.39:0/2441001616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28181a2480 con 0x7f281810b510 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f280fe00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281819cab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f280002a930 tx=0x7f2800031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280003f030 con 0x7f281810b510 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2800004490 con 0x7f281810b510 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28181a2710 con 0x7f281810b510 2024-09-21T01:02:44.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800035070 con 0x7f281810b510 2024-09-21T01:02:44.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.308+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28181a8ed0 con 0x7f281810b510 2024-09-21T01:02:44.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.310+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2818106e50 con 0x7f281810b510 2024-09-21T01:02:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:43 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:43 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2246464438' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T01:02:44.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.314+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2800042030 con 0x7f281810b510 2024-09-21T01:02:44.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.315+0000 7f2814c00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f27fc0b6670 0x7f27fc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:44.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.315+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2800108010 con 0x7f281810b510 2024-09-21T01:02:44.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.315+0000 7f2817400640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f27fc0b6670 0x7f27fc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:44.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.316+0000 7f2817400640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f27fc0b6670 0x7f27fc0b8b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2808009bd0 tx=0x7f2808005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:44.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.318+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f28000d2680 con 0x7f281810b510 2024-09-21T01:02:44.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:43 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:43 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2246464438' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T01:02:44.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.521+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f2818105fe0 con 0x7f281810b510 2024-09-21T01:02:44.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.522+0000 7f2814c00640 1 -- 172.21.15.39:0/2441001616 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2818105fe0 con 0x7f281810b510 2024-09-21T01:02:44.523 INFO:teuthology.orchestra.run.smithi039.stdout:257698037810 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.524+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f27fc0b6670 msgr2=0x7f27fc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.524+0000 7f281cc00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f27fc0b6670 0x7f27fc0b8b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2808009bd0 tx=0x7f2808005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.524+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 msgr2=0x7f281819cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.524+0000 7f281cc00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281819cab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f280002a930 tx=0x7f2800031a20 comp rx=0 tx=0).stop 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 shutdown_connections 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f27fc0b6670 0x7f27fc0b8b80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281810be60 0x7f28181a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f281810b510 0x7f281819cab0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 --2- 172.21.15.39:0/2441001616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2818105bd0 0x7f281819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:44.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 >> 172.21.15.39:0/2441001616 conn(0x7f2818101620 msgr2=0x7f2818103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:44.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 shutdown_connections 2024-09-21T01:02:44.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:44.525+0000 7f281cc00640 1 -- 172.21.15.39:0/2441001616 wait complete. 2024-09-21T01:02:44.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542236 got 60129542239 for osd.1 2024-09-21T01:02:44.698 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:45 smithi150 ceph-mon[34945]: pgmap v366: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2441001616' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T01:02:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:45 smithi160 ceph-mon[34696]: pgmap v366: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2441001616' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T01:02:45.395 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:45 smithi039 ceph-mon[30415]: pgmap v366: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:45.395 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2441001616' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T01:02:45.434 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037806 got 257698037810 for osd.9 2024-09-21T01:02:45.434 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:46.016 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:46.047 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874278 got 279172874282 for osd.10 2024-09-21T01:02:46.047 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:46.077 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:46.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.926+0000 7fdda9600640 1 -- 172.21.15.39:0/431627784 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda402f310 con 0x7fddac109c40 2024-09-21T01:02:46.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 -- 172.21.15.39:0/431627784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac109c40 msgr2=0x7fddac110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:46.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 --2- 172.21.15.39:0/431627784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac109c40 0x7fddac110470 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009ca0 tx=0x7fdda402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:46.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 -- 172.21.15.39:0/431627784 shutdown_connections 2024-09-21T01:02:46.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 --2- 172.21.15.39:0/431627784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac109c40 0x7fddac110470 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:46.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 --2- 172.21.15.39:0/431627784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddac1092f0 0x7fddac109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:46.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 --2- 172.21.15.39:0/431627784 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1039b0 0x7fddac103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:46.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 -- 172.21.15.39:0/431627784 >> 172.21.15.39:0/431627784 conn(0x7fddac0ff440 msgr2=0x7fddac101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:46.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 -- 172.21.15.39:0/431627784 shutdown_connections 2024-09-21T01:02:46.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.928+0000 7fddabe00640 1 -- 172.21.15.39:0/431627784 wait complete. 2024-09-21T01:02:46.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.929+0000 7fddabe00640 1 Processor -- start 2024-09-21T01:02:46.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.929+0000 7fddabe00640 1 -- start start 2024-09-21T01:02:46.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddabe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac1039b0 0x7fddac19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:46.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddabe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1092f0 0x7fddac19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:46.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddabe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddac109c40 0x7fddac19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:46.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddabe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fddac113900 con 0x7fddac1039b0 2024-09-21T01:02:46.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddabe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fddac113780 con 0x7fddac109c40 2024-09-21T01:02:46.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddabe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fddac113a80 con 0x7fddac1092f0 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddaaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac1039b0 0x7fddac19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddaa000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1092f0 0x7fddac19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddaaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac1039b0 0x7fddac19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33534/0 (socket says 172.21.15.39:33534) 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddaaa00640 1 -- 172.21.15.39:0/13920034 learned_addr learned my addr 172.21.15.39:0/13920034 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.930+0000 7fddab400640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddac109c40 0x7fddac19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddaa000640 1 -- 172.21.15.39:0/13920034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddac109c40 msgr2=0x7fddac19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddaa000640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddac109c40 0x7fddac19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddaa000640 1 -- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac1039b0 msgr2=0x7fddac19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddaa000640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac1039b0 0x7fddac19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:46.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddaa000640 1 -- 172.21.15.39:0/13920034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddac1a02c0 con 0x7fddac1092f0 2024-09-21T01:02:46.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddaa000640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1092f0 0x7fddac19a8f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fddac06b990 tx=0x7fdd9800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:46.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9800cd00 con 0x7fddac1092f0 2024-09-21T01:02:46.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddac1a05b0 con 0x7fddac1092f0 2024-09-21T01:02:46.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd980035f0 con 0x7fddac1092f0 2024-09-21T01:02:46.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98010690 con 0x7fddac1092f0 2024-09-21T01:02:46.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.931+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddac1a6d10 con 0x7fddac1092f0 2024-09-21T01:02:46.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.933+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddac103dc0 con 0x7fddac1092f0 2024-09-21T01:02:46.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.936+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fdd980037a0 con 0x7fddac1092f0 2024-09-21T01:02:46.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.937+0000 7fdda3400640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd900b65f0 0x7fdd900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:46.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.937+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdd980e7b80 con 0x7fddac1092f0 2024-09-21T01:02:46.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.938+0000 7fddaaa00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd900b65f0 0x7fdd900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:46.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.938+0000 7fddaaa00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd900b65f0 0x7fdd900b8b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009cd0 tx=0x7fdd9c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:46.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:46.941+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdd980ade10 con 0x7fddac1092f0 2024-09-21T01:02:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.148+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fddac10b4a0 con 0x7fddac1092f0 2024-09-21T01:02:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.149+0000 7fdda3400640 1 -- 172.21.15.39:0/13920034 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdd980b21f0 con 0x7fddac1092f0 2024-09-21T01:02:47.150 INFO:teuthology.orchestra.run.smithi039.stdout:158913790025 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd900b65f0 msgr2=0x7fdd900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd900b65f0 0x7fdd900b8b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009cd0 tx=0x7fdd9c005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1092f0 msgr2=0x7fddac19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1092f0 0x7fddac19a8f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fddac06b990 tx=0x7fdd9800ee50 comp rx=0 tx=0).stop 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 shutdown_connections 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd900b65f0 0x7fdd900b8b00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddac109c40 0x7fddac19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fddac1092f0 0x7fddac19a8f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 --2- 172.21.15.39:0/13920034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddac1039b0 0x7fddac19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 >> 172.21.15.39:0/13920034 conn(0x7fddac0ff440 msgr2=0x7fddac108970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 shutdown_connections 2024-09-21T01:02:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.151+0000 7fddabe00640 1 -- 172.21.15.39:0/13920034 wait complete. 2024-09-21T01:02:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.210+0000 7fba99800640 1 -- 172.21.15.39:0/1671051351 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba941100d0 msgr2=0x7fba94112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.210+0000 7fba99800640 1 --2- 172.21.15.39:0/1671051351 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba941100d0 0x7fba94112510 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009be0 tx=0x7fba8c02a6d0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 -- 172.21.15.39:0/1671051351 shutdown_connections 2024-09-21T01:02:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 --2- 172.21.15.39:0/1671051351 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba941100d0 0x7fba94112510 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 --2- 172.21.15.39:0/1671051351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba94076100 0x7fba9410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 --2- 172.21.15.39:0/1671051351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba940757b0 0x7fba94075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 -- 172.21.15.39:0/1671051351 >> 172.21.15.39:0/1671051351 conn(0x7fba940ff5e0 msgr2=0x7fba94101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 -- 172.21.15.39:0/1671051351 shutdown_connections 2024-09-21T01:02:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.211+0000 7fba99800640 1 -- 172.21.15.39:0/1671051351 wait complete. 2024-09-21T01:02:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.212+0000 7fba99800640 1 Processor -- start 2024-09-21T01:02:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 -- start start 2024-09-21T01:02:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 0x7fba9419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:47.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba94076100 0x7fba9419cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba941100d0 0x7fba941a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fba94115c80 con 0x7fba941100d0 2024-09-21T01:02:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba94115b00 con 0x7fba94076100 2024-09-21T01:02:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.213+0000 7fba99800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fba94115e00 con 0x7fba940757b0 2024-09-21T01:02:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 0x7fba9419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba94076100 0x7fba9419cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba93400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba941100d0 0x7fba941a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba94076100 0x7fba9419cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:33470/0 (socket says 172.21.15.39:33470) 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 0x7fba9419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44868/0 (socket says 172.21.15.39:44868) 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92000640 1 -- 172.21.15.39:0/110032733 learned_addr learned my addr 172.21.15.39:0/110032733 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 -- 172.21.15.39:0/110032733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba94076100 msgr2=0x7fba9419cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba94076100 0x7fba9419cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 -- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba941100d0 msgr2=0x7fba941a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba941100d0 0x7fba941a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.214+0000 7fba92a00640 1 -- 172.21.15.39:0/110032733 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba941a2520 con 0x7fba940757b0 2024-09-21T01:02:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.215+0000 7fba92a00640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 0x7fba9419c5b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fba8400e9f0 tx=0x7fba8400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.215+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8400ccc0 con 0x7fba940757b0 2024-09-21T01:02:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.215+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba840108e0 con 0x7fba940757b0 2024-09-21T01:02:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.215+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84018880 con 0x7fba940757b0 2024-09-21T01:02:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.216+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba941a2810 con 0x7fba940757b0 2024-09-21T01:02:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.216+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba941a8f70 con 0x7fba940757b0 2024-09-21T01:02:47.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.217+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba94075bc0 con 0x7fba940757b0 2024-09-21T01:02:47.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.220+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fba84010430 con 0x7fba940757b0 2024-09-21T01:02:47.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.223+0000 7fba8b400640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba780b6670 0x7fba780b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:47.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.223+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fba840e7cc0 con 0x7fba940757b0 2024-09-21T01:02:47.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.223+0000 7fba92000640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba780b6670 0x7fba780b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:47.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.224+0000 7fba92000640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba780b6670 0x7fba780b8b80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fba94112340 tx=0x7fba80009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:47.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.226+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fba840ee080 con 0x7fba940757b0 2024-09-21T01:02:47.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:47 smithi150 ceph-mon[34945]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:47.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:47 smithi160 ceph-mon[34696]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:47.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:47 smithi039 ceph-mon[30415]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:47.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.428+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fba9402e640 con 0x7fba940757b0 2024-09-21T01:02:47.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.428+0000 7fba8b400640 1 -- 172.21.15.39:0/110032733 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fba840b22d0 con 0x7fba940757b0 2024-09-21T01:02:47.430 INFO:teuthology.orchestra.run.smithi039.stdout:304942678055 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.433+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba780b6670 msgr2=0x7fba780b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.433+0000 7fba99800640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba780b6670 0x7fba780b8b80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fba94112340 tx=0x7fba80009260 comp rx=0 tx=0).stop 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.433+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 msgr2=0x7fba9419c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.433+0000 7fba99800640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 0x7fba9419c5b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fba8400e9f0 tx=0x7fba8400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 shutdown_connections 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba941100d0 0x7fba941a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fba780b6670 0x7fba780b8b80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba94076100 0x7fba9419cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 --2- 172.21.15.39:0/110032733 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba940757b0 0x7fba9419c5b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 >> 172.21.15.39:0/110032733 conn(0x7fba940ff5e0 msgr2=0x7fba94107b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 shutdown_connections 2024-09-21T01:02:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:47.434+0000 7fba99800640 1 -- 172.21.15.39:0/110032733 wait complete. 2024-09-21T01:02:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:48 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/13920034' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T01:02:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:48 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/110032733' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T01:02:48.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:48 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/13920034' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T01:02:48.329 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:48 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/110032733' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T01:02:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:48 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/13920034' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T01:02:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:48 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/110032733' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T01:02:48.367 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678052 got 304942678055 for osd.11 2024-09-21T01:02:48.367 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:48.753 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790023 got 158913790025 for osd.5 2024-09-21T01:02:48.753 DEBUG:teuthology.parallel:result is None 2024-09-21T01:02:48.753 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T01:02:48.753 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:02:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:49 smithi150 ceph-mon[34945]: pgmap v368: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:02:49.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:49 smithi160 ceph-mon[34696]: pgmap v368: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:02:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:49 smithi039 ceph-mon[30415]: pgmap v368: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:02:49.576 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2711335231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 msgr2=0x7fcc040757b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2711335231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc040757b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009860 tx=0x7fcbf402a340 comp rx=0 tx=0).stop 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2711335231 shutdown_connections 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2711335231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 0x7fcc04116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2711335231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc040761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2711335231 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc040757b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2711335231 >> 172.21.15.39:0/2711335231 conn(0x7fcc040ff590 msgr2=0x7fcc041019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2711335231 shutdown_connections 2024-09-21T01:02:50.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.244+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2711335231 wait complete. 2024-09-21T01:02:50.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.246+0000 7fcc0ac00640 1 Processor -- start 2024-09-21T01:02:50.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.246+0000 7fcc0ac00640 1 -- start start 2024-09-21T01:02:50.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc0ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc041a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:50.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc0ac00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc041a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:50.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc0ac00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 0x7fcc041a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:50.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc0ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0411a080 con 0x7fcc04075dd0 2024-09-21T01:02:50.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc0ac00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04119f00 con 0x7fcc04114450 2024-09-21T01:02:50.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc0ac00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0411a200 con 0x7fcc04077380 2024-09-21T01:02:50.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc041a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:50.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc041a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33578/0 (socket says 172.21.15.39:33578) 2024-09-21T01:02:50.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc03e00640 1 -- 172.21.15.39:0/2764432503 learned_addr learned my addr 172.21.15.39:0/2764432503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:50.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc03400640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc041a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:50.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.247+0000 7fcc08c00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 0x7fcc041a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc08c00640 1 -- 172.21.15.39:0/2764432503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 msgr2=0x7fcc041a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc08c00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc041a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc08c00640 1 -- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 msgr2=0x7fcc041a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc08c00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc041a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc08c00640 1 -- 172.21.15.39:0/2764432503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc041a6aa0 con 0x7fcc04114450 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc03400640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc041a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc03e00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc041a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.248+0000 7fcc08c00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 0x7fcc041a6370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009fd0 tx=0x7fcbfc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:50.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.249+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00ce20 con 0x7fcc04114450 2024-09-21T01:02:50.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.249+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc041a6d90 con 0x7fcc04114450 2024-09-21T01:02:50.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.249+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbfc005020 con 0x7fcc04114450 2024-09-21T01:02:50.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.249+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc010690 con 0x7fcc04114450 2024-09-21T01:02:50.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.249+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc041ad4f0 con 0x7fcc04114450 2024-09-21T01:02:50.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.250+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc040761e0 con 0x7fcc04114450 2024-09-21T01:02:50.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.253+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fcbfc0054d0 con 0x7fcc04114450 2024-09-21T01:02:50.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.254+0000 7fcc00c00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcbe80b65f0 0x7fcbe80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:50.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.254+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcbfc0e80c0 con 0x7fcc04114450 2024-09-21T01:02:50.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.255+0000 7fcc03e00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcbe80b65f0 0x7fcbe80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:50.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.255+0000 7fcc03e00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcbe80b65f0 0x7fcbe80b8b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40095f0 tx=0x7fcbf4030340 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:50.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.256+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcbfc0ae2f0 con 0x7fcc04114450 2024-09-21T01:02:50.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.481+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc04063800 con 0x7fcbe80b65f0 2024-09-21T01:02:50.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.488+0000 7fcc00c00640 1 -- 172.21.15.39:0/2764432503 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340359 (secure 0 0 0) 0x7fcc04063800 con 0x7fcbe80b65f0 2024-09-21T01:02:50.493 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:50.495 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:02:50.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcbe80b65f0 msgr2=0x7fcbe80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:50.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcbe80b65f0 0x7fcbe80b8b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40095f0 tx=0x7fcbf4030340 comp rx=0 tx=0).stop 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 msgr2=0x7fcc041a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 0x7fcc041a6370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009fd0 tx=0x7fcbfc00ef10 comp rx=0 tx=0).stop 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 shutdown_connections 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcbe80b65f0 0x7fcbe80b8b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc04114450 0x7fcc041a6370 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc04077380 0x7fcc041a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 --2- 172.21.15.39:0/2764432503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc04075dd0 0x7fcc041a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 >> 172.21.15.39:0/2764432503 conn(0x7fcc040ff590 msgr2=0x7fcc04110350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 shutdown_connections 2024-09-21T01:02:50.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:50.494+0000 7fcc0ac00640 1 -- 172.21.15.39:0/2764432503 wait complete. 2024-09-21T01:02:51.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:51 smithi160 ceph-mon[34696]: pgmap v369: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:51.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:51 smithi160 ceph-mon[34696]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:51.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:51 smithi039 ceph-mon[30415]: pgmap v369: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:51.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:51 smithi039 ceph-mon[30415]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:51.498 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-21T01:02:49.921772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663752,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":516,"num_read_kb":365,"num_write":295,"num_write_kb":2277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371936,"kb_used_data":14300,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569888,"statfs":{"total":1151940427776,"available":1151559565312,"internally_reserved":0,"allocated":14643200,"data_stored":8390354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004214"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489181+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:02:10.489181+0000","last_peered":"2024-09-21T01:02:10.489181+0000","last_clean":"2024-09-21T01:02:10.489181+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:02:10.489181+0000","last_undegraded":"2024-09-21T01:02:10.489181+0000","last_fullsized":"2024-09-21T01:02:10.489181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142862+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:02:11.142862+0000","last_peered":"2024-09-21T01:02:11.142862+0000","last_clean":"2024-09-21T01:02:11.142862+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:02:11.142862+0000","last_undegraded":"2024-09-21T01:02:11.142862+0000","last_fullsized":"2024-09-21T01:02:11.142862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485231+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:02:10.485231+0000","last_peered":"2024-09-21T01:02:10.485231+0000","last_clean":"2024-09-21T01:02:10.485231+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:02:10.485231+0000","last_undegraded":"2024-09-21T01:02:10.485231+0000","last_fullsized":"2024-09-21T01:02:10.485231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.144022+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:02:11.144022+0000","last_peered":"2024-09-21T01:02:11.144022+0000","last_clean":"2024-09-21T01:02:11.144022+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:02:11.144022+0000","last_undegraded":"2024-09-21T01:02:11.144022+0000","last_fullsized":"2024-09-21T01:02:11.144022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645212+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:02:10.645212+0000","last_peered":"2024-09-21T01:02:10.645212+0000","last_clean":"2024-09-21T01:02:10.645212+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:02:10.645212+0000","last_undegraded":"2024-09-21T01:02:10.645212+0000","last_fullsized":"2024-09-21T01:02:10.645212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487212+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:02:10.487212+0000","last_peered":"2024-09-21T01:02:10.487212+0000","last_clean":"2024-09-21T01:02:10.487212+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:02:10.487212+0000","last_undegraded":"2024-09-21T01:02:10.487212+0000","last_fullsized":"2024-09-21T01:02:10.487212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137204+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:02:11.137204+0000","last_peered":"2024-09-21T01:02:11.137204+0000","last_clean":"2024-09-21T01:02:11.137204+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:02:11.137204+0000","last_undegraded":"2024-09-21T01:02:11.137204+0000","last_fullsized":"2024-09-21T01:02:11.137204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964115+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:02:10.964115+0000","last_peered":"2024-09-21T01:02:10.964115+0000","last_clean":"2024-09-21T01:02:10.964115+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:02:10.964115+0000","last_undegraded":"2024-09-21T01:02:10.964115+0000","last_fullsized":"2024-09-21T01:02:10.964115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487333+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:02:10.487333+0000","last_peered":"2024-09-21T01:02:10.487333+0000","last_clean":"2024-09-21T01:02:10.487333+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:02:10.487333+0000","last_undegraded":"2024-09-21T01:02:10.487333+0000","last_fullsized":"2024-09-21T01:02:10.487333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136964+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:02:11.136964+0000","last_peered":"2024-09-21T01:02:11.136964+0000","last_clean":"2024-09-21T01:02:11.136964+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:02:11.136964+0000","last_undegraded":"2024-09-21T01:02:11.136964+0000","last_fullsized":"2024-09-21T01:02:11.136964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489438+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:02:10.489438+0000","last_peered":"2024-09-21T01:02:10.489438+0000","last_clean":"2024-09-21T01:02:10.489438+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:02:10.489438+0000","last_undegraded":"2024-09-21T01:02:10.489438+0000","last_fullsized":"2024-09-21T01:02:10.489438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487284+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:02:10.487284+0000","last_peered":"2024-09-21T01:02:10.487284+0000","last_clean":"2024-09-21T01:02:10.487284+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:02:10.487284+0000","last_undegraded":"2024-09-21T01:02:10.487284+0000","last_fullsized":"2024-09-21T01:02:10.487284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490278+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:02:10.490278+0000","last_peered":"2024-09-21T01:02:10.490278+0000","last_clean":"2024-09-21T01:02:10.490278+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:02:10.490278+0000","last_undegraded":"2024-09-21T01:02:10.490278+0000","last_fullsized":"2024-09-21T01:02:10.490278+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136801+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:02:11.136801+0000","last_peered":"2024-09-21T01:02:11.136801+0000","last_clean":"2024-09-21T01:02:11.136801+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:02:11.136801+0000","last_undegraded":"2024-09-21T01:02:11.136801+0000","last_fullsized":"2024-09-21T01:02:11.136801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487975+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:02:10.487975+0000","last_peered":"2024-09-21T01:02:10.487975+0000","last_clean":"2024-09-21T01:02:10.487975+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:02:10.487975+0000","last_undegraded":"2024-09-21T01:02:10.487975+0000","last_fullsized":"2024-09-21T01:02:10.487975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143467+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:02:11.143467+0000","last_peered":"2024-09-21T01:02:11.143467+0000","last_clean":"2024-09-21T01:02:11.143467+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:02:11.143467+0000","last_undegraded":"2024-09-21T01:02:11.143467+0000","last_fullsized":"2024-09-21T01:02:11.143467+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485342+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:02:10.485342+0000","last_peered":"2024-09-21T01:02:10.485342+0000","last_clean":"2024-09-21T01:02:10.485342+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:02:10.485342+0000","last_undegraded":"2024-09-21T01:02:10.485342+0000","last_fullsized":"2024-09-21T01:02:10.485342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487888+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:02:10.487888+0000","last_peered":"2024-09-21T01:02:10.487888+0000","last_clean":"2024-09-21T01:02:10.487888+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:02:10.487888+0000","last_undegraded":"2024-09-21T01:02:10.487888+0000","last_fullsized":"2024-09-21T01:02:10.487888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316462+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:02:11.316462+0000","last_peered":"2024-09-21T01:02:11.316462+0000","last_clean":"2024-09-21T01:02:11.316462+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:02:11.316462+0000","last_undegraded":"2024-09-21T01:02:11.316462+0000","last_fullsized":"2024-09-21T01:02:11.316462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143293+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:02:11.143293+0000","last_peered":"2024-09-21T01:02:11.143293+0000","last_clean":"2024-09-21T01:02:11.143293+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:02:11.143293+0000","last_undegraded":"2024-09-21T01:02:11.143293+0000","last_fullsized":"2024-09-21T01:02:11.143293+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488334+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:02:10.488334+0000","last_peered":"2024-09-21T01:02:10.488334+0000","last_clean":"2024-09-21T01:02:10.488334+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:02:10.488334+0000","last_undegraded":"2024-09-21T01:02:10.488334+0000","last_fullsized":"2024-09-21T01:02:10.488334+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136709+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:02:11.136709+0000","last_peered":"2024-09-21T01:02:11.136709+0000","last_clean":"2024-09-21T01:02:11.136709+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:02:11.136709+0000","last_undegraded":"2024-09-21T01:02:11.136709+0000","last_fullsized":"2024-09-21T01:02:11.136709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964583+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:02:10.964583+0000","last_peered":"2024-09-21T01:02:10.964583+0000","last_clean":"2024-09-21T01:02:10.964583+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:02:10.964583+0000","last_undegraded":"2024-09-21T01:02:10.964583+0000","last_fullsized":"2024-09-21T01:02:10.964583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488036+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:02:10.488036+0000","last_peered":"2024-09-21T01:02:10.488036+0000","last_clean":"2024-09-21T01:02:10.488036+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:02:10.488036+0000","last_undegraded":"2024-09-21T01:02:10.488036+0000","last_fullsized":"2024-09-21T01:02:10.488036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645873+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:02:10.645873+0000","last_peered":"2024-09-21T01:02:10.645873+0000","last_clean":"2024-09-21T01:02:10.645873+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:02:10.645873+0000","last_undegraded":"2024-09-21T01:02:10.645873+0000","last_fullsized":"2024-09-21T01:02:10.645873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488374+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:02:10.488374+0000","last_peered":"2024-09-21T01:02:10.488374+0000","last_clean":"2024-09-21T01:02:10.488374+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:02:10.488374+0000","last_undegraded":"2024-09-21T01:02:10.488374+0000","last_fullsized":"2024-09-21T01:02:10.488374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485574+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:02:10.485574+0000","last_peered":"2024-09-21T01:02:10.485574+0000","last_clean":"2024-09-21T01:02:10.485574+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:02:10.485574+0000","last_undegraded":"2024-09-21T01:02:10.485574+0000","last_fullsized":"2024-09-21T01:02:10.485574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142902+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:02:11.142902+0000","last_peered":"2024-09-21T01:02:11.142902+0000","last_clean":"2024-09-21T01:02:11.142902+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:02:11.142902+0000","last_undegraded":"2024-09-21T01:02:11.142902+0000","last_fullsized":"2024-09-21T01:02:11.142902+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488972+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:02:10.488972+0000","last_peered":"2024-09-21T01:02:10.488972+0000","last_clean":"2024-09-21T01:02:10.488972+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:02:10.488972+0000","last_undegraded":"2024-09-21T01:02:10.488972+0000","last_fullsized":"2024-09-21T01:02:10.488972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142917+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:02:11.142917+0000","last_peered":"2024-09-21T01:02:11.142917+0000","last_clean":"2024-09-21T01:02:11.142917+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:02:11.142917+0000","last_undegraded":"2024-09-21T01:02:11.142917+0000","last_fullsized":"2024-09-21T01:02:11.142917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484965+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:02:10.484965+0000","last_peered":"2024-09-21T01:02:10.484965+0000","last_clean":"2024-09-21T01:02:10.484965+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:02:10.484965+0000","last_undegraded":"2024-09-21T01:02:10.484965+0000","last_fullsized":"2024-09-21T01:02:10.484965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137262+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:02:11.137262+0000","last_peered":"2024-09-21T01:02:11.137262+0000","last_clean":"2024-09-21T01:02:11.137262+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:02:11.137262+0000","last_undegraded":"2024-09-21T01:02:11.137262+0000","last_fullsized":"2024-09-21T01:02:11.137262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316884+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:02:11.316884+0000","last_peered":"2024-09-21T01:02:11.316884+0000","last_clean":"2024-09-21T01:02:11.316884+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:02:11.316884+0000","last_undegraded":"2024-09-21T01:02:11.316884+0000","last_fullsized":"2024-09-21T01:02:11.316884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517184+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:02:10.517184+0000","last_peered":"2024-09-21T01:02:10.517184+0000","last_clean":"2024-09-21T01:02:10.517184+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:02:10.517184+0000","last_undegraded":"2024-09-21T01:02:10.517184+0000","last_fullsized":"2024-09-21T01:02:10.517184+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488662+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:02:10.488662+0000","last_peered":"2024-09-21T01:02:10.488662+0000","last_clean":"2024-09-21T01:02:10.488662+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:02:10.488662+0000","last_undegraded":"2024-09-21T01:02:10.488662+0000","last_fullsized":"2024-09-21T01:02:10.488662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136488+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:02:11.136488+0000","last_peered":"2024-09-21T01:02:11.136488+0000","last_clean":"2024-09-21T01:02:11.136488+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:02:11.136488+0000","last_undegraded":"2024-09-21T01:02:11.136488+0000","last_fullsized":"2024-09-21T01:02:11.136488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517079+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:02:10.517079+0000","last_peered":"2024-09-21T01:02:10.517079+0000","last_clean":"2024-09-21T01:02:10.517079+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:02:10.517079+0000","last_undegraded":"2024-09-21T01:02:10.517079+0000","last_fullsized":"2024-09-21T01:02:10.517079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490887+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:02:10.490887+0000","last_peered":"2024-09-21T01:02:10.490887+0000","last_clean":"2024-09-21T01:02:10.490887+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:02:10.490887+0000","last_undegraded":"2024-09-21T01:02:10.490887+0000","last_fullsized":"2024-09-21T01:02:10.490887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484758+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:02:10.484758+0000","last_peered":"2024-09-21T01:02:10.484758+0000","last_clean":"2024-09-21T01:02:10.484758+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:02:10.484758+0000","last_undegraded":"2024-09-21T01:02:10.484758+0000","last_fullsized":"2024-09-21T01:02:10.484758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489249+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:02:10.489249+0000","last_peered":"2024-09-21T01:02:10.489249+0000","last_clean":"2024-09-21T01:02:10.489249+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:02:10.489249+0000","last_undegraded":"2024-09-21T01:02:10.489249+0000","last_fullsized":"2024-09-21T01:02:10.489249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:46.491081+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:02:46.491081+0000","last_peered":"2024-09-21T01:02:46.491081+0000","last_clean":"2024-09-21T01:02:46.491081+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:02:46.491081+0000","last_undegraded":"2024-09-21T01:02:46.491081+0000","last_fullsized":"2024-09-21T01:02:46.491081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485032+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:02:10.485032+0000","last_peered":"2024-09-21T01:02:10.485032+0000","last_clean":"2024-09-21T01:02:10.485032+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:02:10.485032+0000","last_undegraded":"2024-09-21T01:02:10.485032+0000","last_fullsized":"2024-09-21T01:02:10.485032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484255+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:02:10.484255+0000","last_peered":"2024-09-21T01:02:10.484255+0000","last_clean":"2024-09-21T01:02:10.484255+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:02:10.484255+0000","last_undegraded":"2024-09-21T01:02:10.484255+0000","last_fullsized":"2024-09-21T01:02:10.484255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490838+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:02:10.490838+0000","last_peered":"2024-09-21T01:02:10.490838+0000","last_clean":"2024-09-21T01:02:10.490838+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:02:10.490838+0000","last_undegraded":"2024-09-21T01:02:10.490838+0000","last_fullsized":"2024-09-21T01:02:10.490838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489919+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:02:10.489919+0000","last_peered":"2024-09-21T01:02:10.489919+0000","last_clean":"2024-09-21T01:02:10.489919+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:02:10.489919+0000","last_undegraded":"2024-09-21T01:02:10.489919+0000","last_fullsized":"2024-09-21T01:02:10.489919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143430+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:02:11.143430+0000","last_peered":"2024-09-21T01:02:11.143430+0000","last_clean":"2024-09-21T01:02:11.143430+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:02:11.143430+0000","last_undegraded":"2024-09-21T01:02:11.143430+0000","last_fullsized":"2024-09-21T01:02:11.143430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487931+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:02:10.487931+0000","last_peered":"2024-09-21T01:02:10.487931+0000","last_clean":"2024-09-21T01:02:10.487931+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:02:10.487931+0000","last_undegraded":"2024-09-21T01:02:10.487931+0000","last_fullsized":"2024-09-21T01:02:10.487931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490885+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:02:10.490885+0000","last_peered":"2024-09-21T01:02:10.490885+0000","last_clean":"2024-09-21T01:02:10.490885+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:02:10.490885+0000","last_undegraded":"2024-09-21T01:02:10.490885+0000","last_fullsized":"2024-09-21T01:02:10.490885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490147+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:02:10.490147+0000","last_peered":"2024-09-21T01:02:10.490147+0000","last_clean":"2024-09-21T01:02:10.490147+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:02:10.490147+0000","last_undegraded":"2024-09-21T01:02:10.490147+0000","last_fullsized":"2024-09-21T01:02:10.490147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490730+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:02:10.490730+0000","last_peered":"2024-09-21T01:02:10.490730+0000","last_clean":"2024-09-21T01:02:10.490730+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:02:10.490730+0000","last_undegraded":"2024-09-21T01:02:10.490730+0000","last_fullsized":"2024-09-21T01:02:10.490730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485884+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:02:10.485884+0000","last_peered":"2024-09-21T01:02:10.485884+0000","last_clean":"2024-09-21T01:02:10.485884+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:02:10.485884+0000","last_undegraded":"2024-09-21T01:02:10.485884+0000","last_fullsized":"2024-09-21T01:02:10.485884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964535+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:02:10.964535+0000","last_peered":"2024-09-21T01:02:10.964535+0000","last_clean":"2024-09-21T01:02:10.964535+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:02:10.964535+0000","last_undegraded":"2024-09-21T01:02:10.964535+0000","last_fullsized":"2024-09-21T01:02:10.964535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490480+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:02:10.490480+0000","last_peered":"2024-09-21T01:02:10.490480+0000","last_clean":"2024-09-21T01:02:10.490480+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:02:10.490480+0000","last_undegraded":"2024-09-21T01:02:10.490480+0000","last_fullsized":"2024-09-21T01:02:10.490480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489782+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:02:10.489782+0000","last_peered":"2024-09-21T01:02:10.489782+0000","last_clean":"2024-09-21T01:02:10.489782+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:02:10.489782+0000","last_undegraded":"2024-09-21T01:02:10.489782+0000","last_fullsized":"2024-09-21T01:02:10.489782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645520+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:02:10.645520+0000","last_peered":"2024-09-21T01:02:10.645520+0000","last_clean":"2024-09-21T01:02:10.645520+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:02:10.645520+0000","last_undegraded":"2024-09-21T01:02:10.645520+0000","last_fullsized":"2024-09-21T01:02:10.645520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490362+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:02:10.490362+0000","last_peered":"2024-09-21T01:02:10.490362+0000","last_clean":"2024-09-21T01:02:10.490362+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:02:10.490362+0000","last_undegraded":"2024-09-21T01:02:10.490362+0000","last_fullsized":"2024-09-21T01:02:10.490362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485031+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:02:10.485031+0000","last_peered":"2024-09-21T01:02:10.485031+0000","last_clean":"2024-09-21T01:02:10.485031+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:02:10.485031+0000","last_undegraded":"2024-09-21T01:02:10.485031+0000","last_fullsized":"2024-09-21T01:02:10.485031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488992+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:02:10.488992+0000","last_peered":"2024-09-21T01:02:10.488992+0000","last_clean":"2024-09-21T01:02:10.488992+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:02:10.488992+0000","last_undegraded":"2024-09-21T01:02:10.488992+0000","last_fullsized":"2024-09-21T01:02:10.488992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316507+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:02:11.316507+0000","last_peered":"2024-09-21T01:02:11.316507+0000","last_clean":"2024-09-21T01:02:11.316507+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:02:11.316507+0000","last_undegraded":"2024-09-21T01:02:11.316507+0000","last_fullsized":"2024-09-21T01:02:11.316507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317248+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:02:11.317248+0000","last_peered":"2024-09-21T01:02:11.317248+0000","last_clean":"2024-09-21T01:02:11.317248+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:02:11.317248+0000","last_undegraded":"2024-09-21T01:02:11.317248+0000","last_fullsized":"2024-09-21T01:02:11.317248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489724+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:02:10.489724+0000","last_peered":"2024-09-21T01:02:10.489724+0000","last_clean":"2024-09-21T01:02:10.489724+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:02:10.489724+0000","last_undegraded":"2024-09-21T01:02:10.489724+0000","last_fullsized":"2024-09-21T01:02:10.489724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.966099+0000","last_change":"2024-09-21T01:02:10.966004+0000","last_active":"2024-09-21T01:02:10.966099+0000","last_peered":"2024-09-21T01:02:10.966099+0000","last_clean":"2024-09-21T01:02:10.966099+0000","last_became_active":"2024-09-21T01:02:10.965645+0000","last_became_peered":"2024-09-21T01:02:10.965645+0000","last_unstale":"2024-09-21T01:02:10.966099+0000","last_undegraded":"2024-09-21T01:02:10.966099+0000","last_fullsized":"2024-09-21T01:02:10.966099+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:12.070017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136545+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:02:11.136545+0000","last_peered":"2024-09-21T01:02:11.136545+0000","last_clean":"2024-09-21T01:02:11.136545+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:02:11.136545+0000","last_undegraded":"2024-09-21T01:02:11.136545+0000","last_fullsized":"2024-09-21T01:02:11.136545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484951+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:02:10.484951+0000","last_peered":"2024-09-21T01:02:10.484951+0000","last_clean":"2024-09-21T01:02:10.484951+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:02:10.484951+0000","last_undegraded":"2024-09-21T01:02:10.484951+0000","last_fullsized":"2024-09-21T01:02:10.484951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489976+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:02:10.489976+0000","last_peered":"2024-09-21T01:02:10.489976+0000","last_clean":"2024-09-21T01:02:10.489976+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:02:10.489976+0000","last_undegraded":"2024-09-21T01:02:10.489976+0000","last_fullsized":"2024-09-21T01:02:10.489976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:45.490105+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:02:45.490105+0000","last_peered":"2024-09-21T01:02:45.490105+0000","last_clean":"2024-09-21T01:02:45.490105+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:02:45.490105+0000","last_undegraded":"2024-09-21T01:02:45.490105+0000","last_fullsized":"2024-09-21T01:02:45.490105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489100+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:02:10.489100+0000","last_peered":"2024-09-21T01:02:10.489100+0000","last_clean":"2024-09-21T01:02:10.489100+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:02:10.489100+0000","last_undegraded":"2024-09-21T01:02:10.489100+0000","last_fullsized":"2024-09-21T01:02:10.489100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490595+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:02:10.490595+0000","last_peered":"2024-09-21T01:02:10.490595+0000","last_clean":"2024-09-21T01:02:10.490595+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:02:10.490595+0000","last_undegraded":"2024-09-21T01:02:10.490595+0000","last_fullsized":"2024-09-21T01:02:10.490595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489394+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:02:10.489394+0000","last_peered":"2024-09-21T01:02:10.489394+0000","last_clean":"2024-09-21T01:02:10.489394+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:02:10.489394+0000","last_undegraded":"2024-09-21T01:02:10.489394+0000","last_fullsized":"2024-09-21T01:02:10.489394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316850+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:02:11.316850+0000","last_peered":"2024-09-21T01:02:11.316850+0000","last_clean":"2024-09-21T01:02:11.316850+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:02:11.316850+0000","last_undegraded":"2024-09-21T01:02:11.316850+0000","last_fullsized":"2024-09-21T01:02:11.316850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484391+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:02:10.484391+0000","last_peered":"2024-09-21T01:02:10.484391+0000","last_clean":"2024-09-21T01:02:10.484391+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:02:10.484391+0000","last_undegraded":"2024-09-21T01:02:10.484391+0000","last_fullsized":"2024-09-21T01:02:10.484391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485499+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:02:10.485499+0000","last_peered":"2024-09-21T01:02:10.485499+0000","last_clean":"2024-09-21T01:02:10.485499+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:02:10.485499+0000","last_undegraded":"2024-09-21T01:02:10.485499+0000","last_fullsized":"2024-09-21T01:02:10.485499+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489623+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:02:10.489623+0000","last_peered":"2024-09-21T01:02:10.489623+0000","last_clean":"2024-09-21T01:02:10.489623+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:02:10.489623+0000","last_undegraded":"2024-09-21T01:02:10.489623+0000","last_fullsized":"2024-09-21T01:02:10.489623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.135886+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:02:11.135886+0000","last_peered":"2024-09-21T01:02:11.135886+0000","last_clean":"2024-09-21T01:02:11.135886+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:02:11.135886+0000","last_undegraded":"2024-09-21T01:02:11.135886+0000","last_fullsized":"2024-09-21T01:02:11.135886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":227,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:49.485433+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:02:49.485433+0000","last_peered":"2024-09-21T01:02:49.485433+0000","last_clean":"2024-09-21T01:02:49.485433+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:02:49.485433+0000","last_undegraded":"2024-09-21T01:02:49.485433+0000","last_fullsized":"2024-09-21T01:02:49.485433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":148,"num_read_kb":141,"num_write":36,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488137+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:02:10.488137+0000","last_peered":"2024-09-21T01:02:10.488137+0000","last_clean":"2024-09-21T01:02:10.488137+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:02:10.488137+0000","last_undegraded":"2024-09-21T01:02:10.488137+0000","last_fullsized":"2024-09-21T01:02:10.488137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964114+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:02:10.964114+0000","last_peered":"2024-09-21T01:02:10.964114+0000","last_clean":"2024-09-21T01:02:10.964114+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:02:10.964114+0000","last_undegraded":"2024-09-21T01:02:10.964114+0000","last_fullsized":"2024-09-21T01:02:10.964114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489715+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:02:10.489715+0000","last_peered":"2024-09-21T01:02:10.489715+0000","last_clean":"2024-09-21T01:02:10.489715+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:02:10.489715+0000","last_undegraded":"2024-09-21T01:02:10.489715+0000","last_fullsized":"2024-09-21T01:02:10.489715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487635+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:02:10.487635+0000","last_peered":"2024-09-21T01:02:10.487635+0000","last_clean":"2024-09-21T01:02:10.487635+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:02:10.487635+0000","last_undegraded":"2024-09-21T01:02:10.487635+0000","last_fullsized":"2024-09-21T01:02:10.487635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.644982+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:02:10.644982+0000","last_peered":"2024-09-21T01:02:10.644982+0000","last_clean":"2024-09-21T01:02:10.644982+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:02:10.644982+0000","last_undegraded":"2024-09-21T01:02:10.644982+0000","last_fullsized":"2024-09-21T01:02:10.644982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490215+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:02:10.490215+0000","last_peered":"2024-09-21T01:02:10.490215+0000","last_clean":"2024-09-21T01:02:10.490215+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:02:10.490215+0000","last_undegraded":"2024-09-21T01:02:10.490215+0000","last_fullsized":"2024-09-21T01:02:10.490215+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487904+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:02:10.487904+0000","last_peered":"2024-09-21T01:02:10.487904+0000","last_clean":"2024-09-21T01:02:10.487904+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:02:10.487904+0000","last_undegraded":"2024-09-21T01:02:10.487904+0000","last_fullsized":"2024-09-21T01:02:10.487904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964130+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:02:10.964130+0000","last_peered":"2024-09-21T01:02:10.964130+0000","last_clean":"2024-09-21T01:02:10.964130+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:02:10.964130+0000","last_undegraded":"2024-09-21T01:02:10.964130+0000","last_fullsized":"2024-09-21T01:02:10.964130+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:48.483835+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:02:48.483835+0000","last_peered":"2024-09-21T01:02:48.483835+0000","last_clean":"2024-09-21T01:02:48.483835+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:02:48.483835+0000","last_undegraded":"2024-09-21T01:02:48.483835+0000","last_fullsized":"2024-09-21T01:02:48.483835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143060+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:02:11.143060+0000","last_peered":"2024-09-21T01:02:11.143060+0000","last_clean":"2024-09-21T01:02:11.143060+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:02:11.143060+0000","last_undegraded":"2024-09-21T01:02:11.143060+0000","last_fullsized":"2024-09-21T01:02:11.143060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645694+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:02:10.645694+0000","last_peered":"2024-09-21T01:02:10.645694+0000","last_clean":"2024-09-21T01:02:10.645694+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:02:10.645694+0000","last_undegraded":"2024-09-21T01:02:10.645694+0000","last_fullsized":"2024-09-21T01:02:10.645694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488122+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:02:10.488122+0000","last_peered":"2024-09-21T01:02:10.488122+0000","last_clean":"2024-09-21T01:02:10.488122+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:02:10.488122+0000","last_undegraded":"2024-09-21T01:02:10.488122+0000","last_fullsized":"2024-09-21T01:02:10.488122+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489363+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:02:10.489363+0000","last_peered":"2024-09-21T01:02:10.489363+0000","last_clean":"2024-09-21T01:02:10.489363+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:02:10.489363+0000","last_undegraded":"2024-09-21T01:02:10.489363+0000","last_fullsized":"2024-09-21T01:02:10.489363+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317103+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:02:11.317103+0000","last_peered":"2024-09-21T01:02:11.317103+0000","last_clean":"2024-09-21T01:02:11.317103+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:02:11.317103+0000","last_undegraded":"2024-09-21T01:02:11.317103+0000","last_fullsized":"2024-09-21T01:02:11.317103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142654+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:02:11.142654+0000","last_peered":"2024-09-21T01:02:11.142654+0000","last_clean":"2024-09-21T01:02:11.142654+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:02:11.142654+0000","last_undegraded":"2024-09-21T01:02:11.142654+0000","last_fullsized":"2024-09-21T01:02:11.142654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484193+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:02:10.484193+0000","last_peered":"2024-09-21T01:02:10.484193+0000","last_clean":"2024-09-21T01:02:10.484193+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:02:10.484193+0000","last_undegraded":"2024-09-21T01:02:10.484193+0000","last_fullsized":"2024-09-21T01:02:10.484193+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517512+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:02:10.517512+0000","last_peered":"2024-09-21T01:02:10.517512+0000","last_clean":"2024-09-21T01:02:10.517512+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:02:10.517512+0000","last_undegraded":"2024-09-21T01:02:10.517512+0000","last_fullsized":"2024-09-21T01:02:10.517512+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489999+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:02:10.489999+0000","last_peered":"2024-09-21T01:02:10.489999+0000","last_clean":"2024-09-21T01:02:10.489999+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:02:10.489999+0000","last_undegraded":"2024-09-21T01:02:10.489999+0000","last_fullsized":"2024-09-21T01:02:10.489999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485566+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:02:10.485566+0000","last_peered":"2024-09-21T01:02:10.485566+0000","last_clean":"2024-09-21T01:02:10.485566+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:02:10.485566+0000","last_undegraded":"2024-09-21T01:02:10.485566+0000","last_fullsized":"2024-09-21T01:02:10.485566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:45.480143+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:02:45.480143+0000","last_peered":"2024-09-21T01:02:45.480143+0000","last_clean":"2024-09-21T01:02:45.480143+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:02:45.480143+0000","last_undegraded":"2024-09-21T01:02:45.480143+0000","last_fullsized":"2024-09-21T01:02:45.480143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.135852+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:02:11.135852+0000","last_peered":"2024-09-21T01:02:11.135852+0000","last_clean":"2024-09-21T01:02:11.135852+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:02:11.135852+0000","last_undegraded":"2024-09-21T01:02:11.135852+0000","last_fullsized":"2024-09-21T01:02:11.135852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485531+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:02:10.485531+0000","last_peered":"2024-09-21T01:02:10.485531+0000","last_clean":"2024-09-21T01:02:10.485531+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:02:10.485531+0000","last_undegraded":"2024-09-21T01:02:10.485531+0000","last_fullsized":"2024-09-21T01:02:10.485531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316525+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:02:11.316525+0000","last_peered":"2024-09-21T01:02:11.316525+0000","last_clean":"2024-09-21T01:02:11.316525+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:02:11.316525+0000","last_undegraded":"2024-09-21T01:02:11.316525+0000","last_fullsized":"2024-09-21T01:02:11.316525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485868+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:02:10.485868+0000","last_peered":"2024-09-21T01:02:10.485868+0000","last_clean":"2024-09-21T01:02:10.485868+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:02:10.485868+0000","last_undegraded":"2024-09-21T01:02:10.485868+0000","last_fullsized":"2024-09-21T01:02:10.485868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485364+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:02:10.485364+0000","last_peered":"2024-09-21T01:02:10.485364+0000","last_clean":"2024-09-21T01:02:10.485364+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:02:10.485364+0000","last_undegraded":"2024-09-21T01:02:10.485364+0000","last_fullsized":"2024-09-21T01:02:10.485364+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645189+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:02:10.645189+0000","last_peered":"2024-09-21T01:02:10.645189+0000","last_clean":"2024-09-21T01:02:10.645189+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:02:10.645189+0000","last_undegraded":"2024-09-21T01:02:10.645189+0000","last_fullsized":"2024-09-21T01:02:10.645189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.647189+0000","last_change":"2024-09-21T01:02:10.647066+0000","last_active":"2024-09-21T01:02:10.647189+0000","last_peered":"2024-09-21T01:02:10.647189+0000","last_clean":"2024-09-21T01:02:10.647189+0000","last_became_active":"2024-09-21T01:02:10.646487+0000","last_became_peered":"2024-09-21T01:02:10.646487+0000","last_unstale":"2024-09-21T01:02:10.647189+0000","last_undegraded":"2024-09-21T01:02:10.647189+0000","last_fullsized":"2024-09-21T01:02:10.647189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:19:28.445094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316477+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:02:11.316477+0000","last_peered":"2024-09-21T01:02:11.316477+0000","last_clean":"2024-09-21T01:02:11.316477+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:02:11.316477+0000","last_undegraded":"2024-09-21T01:02:11.316477+0000","last_fullsized":"2024-09-21T01:02:11.316477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485777+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:02:10.485777+0000","last_peered":"2024-09-21T01:02:10.485777+0000","last_clean":"2024-09-21T01:02:10.485777+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:02:10.485777+0000","last_undegraded":"2024-09-21T01:02:10.485777+0000","last_fullsized":"2024-09-21T01:02:10.485777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142910+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:02:11.142910+0000","last_peered":"2024-09-21T01:02:11.142910+0000","last_clean":"2024-09-21T01:02:11.142910+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:02:11.142910+0000","last_undegraded":"2024-09-21T01:02:11.142910+0000","last_fullsized":"2024-09-21T01:02:11.142910+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489735+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:02:10.489735+0000","last_peered":"2024-09-21T01:02:10.489735+0000","last_clean":"2024-09-21T01:02:10.489735+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:02:10.489735+0000","last_undegraded":"2024-09-21T01:02:10.489735+0000","last_fullsized":"2024-09-21T01:02:10.489735+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.318874+0000","last_change":"2024-09-21T01:02:11.318295+0000","last_active":"2024-09-21T01:02:11.318874+0000","last_peered":"2024-09-21T01:02:11.318874+0000","last_clean":"2024-09-21T01:02:11.318874+0000","last_became_active":"2024-09-21T01:02:11.317912+0000","last_became_peered":"2024-09-21T01:02:11.317912+0000","last_unstale":"2024-09-21T01:02:11.318874+0000","last_undegraded":"2024-09-21T01:02:11.318874+0000","last_fullsized":"2024-09-21T01:02:11.318874+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:53:47.063340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964148+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:02:10.964148+0000","last_peered":"2024-09-21T01:02:10.964148+0000","last_clean":"2024-09-21T01:02:10.964148+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:02:10.964148+0000","last_undegraded":"2024-09-21T01:02:10.964148+0000","last_fullsized":"2024-09-21T01:02:10.964148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484627+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:02:10.484627+0000","last_peered":"2024-09-21T01:02:10.484627+0000","last_clean":"2024-09-21T01:02:10.484627+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:02:10.484627+0000","last_undegraded":"2024-09-21T01:02:10.484627+0000","last_fullsized":"2024-09-21T01:02:10.484627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316477+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:02:11.316477+0000","last_peered":"2024-09-21T01:02:11.316477+0000","last_clean":"2024-09-21T01:02:11.316477+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:02:11.316477+0000","last_undegraded":"2024-09-21T01:02:11.316477+0000","last_fullsized":"2024-09-21T01:02:11.316477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485672+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:02:10.485672+0000","last_peered":"2024-09-21T01:02:10.485672+0000","last_clean":"2024-09-21T01:02:10.485672+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:02:10.485672+0000","last_undegraded":"2024-09-21T01:02:10.485672+0000","last_fullsized":"2024-09-21T01:02:10.485672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488903+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:02:10.488903+0000","last_peered":"2024-09-21T01:02:10.488903+0000","last_clean":"2024-09-21T01:02:10.488903+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:02:10.488903+0000","last_undegraded":"2024-09-21T01:02:10.488903+0000","last_fullsized":"2024-09-21T01:02:10.488903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317334+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:02:11.317334+0000","last_peered":"2024-09-21T01:02:11.317334+0000","last_clean":"2024-09-21T01:02:11.317334+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:02:11.317334+0000","last_undegraded":"2024-09-21T01:02:11.317334+0000","last_fullsized":"2024-09-21T01:02:11.317334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964029+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:02:10.964029+0000","last_peered":"2024-09-21T01:02:10.964029+0000","last_clean":"2024-09-21T01:02:10.964029+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:02:10.964029+0000","last_undegraded":"2024-09-21T01:02:10.964029+0000","last_fullsized":"2024-09-21T01:02:10.964029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.644922+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:02:10.644922+0000","last_peered":"2024-09-21T01:02:10.644922+0000","last_clean":"2024-09-21T01:02:10.644922+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:02:10.644922+0000","last_undegraded":"2024-09-21T01:02:10.644922+0000","last_fullsized":"2024-09-21T01:02:10.644922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645391+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:02:10.645391+0000","last_peered":"2024-09-21T01:02:10.645391+0000","last_clean":"2024-09-21T01:02:10.645391+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:02:10.645391+0000","last_undegraded":"2024-09-21T01:02:10.645391+0000","last_fullsized":"2024-09-21T01:02:10.645391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484591+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:02:10.484591+0000","last_peered":"2024-09-21T01:02:10.484591+0000","last_clean":"2024-09-21T01:02:10.484591+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:02:10.484591+0000","last_undegraded":"2024-09-21T01:02:10.484591+0000","last_fullsized":"2024-09-21T01:02:10.484591+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488141+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:02:10.488141+0000","last_peered":"2024-09-21T01:02:10.488141+0000","last_clean":"2024-09-21T01:02:10.488141+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:02:10.488141+0000","last_undegraded":"2024-09-21T01:02:10.488141+0000","last_fullsized":"2024-09-21T01:02:10.488141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490320+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:02:10.490320+0000","last_peered":"2024-09-21T01:02:10.490320+0000","last_clean":"2024-09-21T01:02:10.490320+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:02:10.490320+0000","last_undegraded":"2024-09-21T01:02:10.490320+0000","last_fullsized":"2024-09-21T01:02:10.490320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517008+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:02:10.517008+0000","last_peered":"2024-09-21T01:02:10.517008+0000","last_clean":"2024-09-21T01:02:10.517008+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:02:10.517008+0000","last_undegraded":"2024-09-21T01:02:10.517008+0000","last_fullsized":"2024-09-21T01:02:10.517008+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964290+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:02:10.964290+0000","last_peered":"2024-09-21T01:02:10.964290+0000","last_clean":"2024-09-21T01:02:10.964290+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:02:10.964290+0000","last_undegraded":"2024-09-21T01:02:10.964290+0000","last_fullsized":"2024-09-21T01:02:10.964290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142650+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:02:11.142650+0000","last_peered":"2024-09-21T01:02:11.142650+0000","last_clean":"2024-09-21T01:02:11.142650+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:02:11.142650+0000","last_undegraded":"2024-09-21T01:02:11.142650+0000","last_fullsized":"2024-09-21T01:02:11.142650+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.138407+0000","last_change":"2024-09-21T01:02:11.138284+0000","last_active":"2024-09-21T01:02:11.138407+0000","last_peered":"2024-09-21T01:02:11.138407+0000","last_clean":"2024-09-21T01:02:11.138407+0000","last_became_active":"2024-09-21T01:02:11.137782+0000","last_became_peered":"2024-09-21T01:02:11.137782+0000","last_unstale":"2024-09-21T01:02:11.138407+0000","last_undegraded":"2024-09-21T01:02:11.138407+0000","last_fullsized":"2024-09-21T01:02:11.138407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:57.753015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.516933+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:02:10.516933+0000","last_peered":"2024-09-21T01:02:10.516933+0000","last_clean":"2024-09-21T01:02:10.516933+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:02:10.516933+0000","last_undegraded":"2024-09-21T01:02:10.516933+0000","last_fullsized":"2024-09-21T01:02:10.516933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.135836+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:02:11.135836+0000","last_peered":"2024-09-21T01:02:11.135836+0000","last_clean":"2024-09-21T01:02:11.135836+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:02:11.135836+0000","last_undegraded":"2024-09-21T01:02:11.135836+0000","last_fullsized":"2024-09-21T01:02:11.135836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488426+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:02:10.488426+0000","last_peered":"2024-09-21T01:02:10.488426+0000","last_clean":"2024-09-21T01:02:10.488426+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:02:10.488426+0000","last_undegraded":"2024-09-21T01:02:10.488426+0000","last_fullsized":"2024-09-21T01:02:10.488426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645067+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:02:10.645067+0000","last_peered":"2024-09-21T01:02:10.645067+0000","last_clean":"2024-09-21T01:02:10.645067+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:02:10.645067+0000","last_undegraded":"2024-09-21T01:02:10.645067+0000","last_fullsized":"2024-09-21T01:02:10.645067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645894+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:02:10.645894+0000","last_peered":"2024-09-21T01:02:10.645894+0000","last_clean":"2024-09-21T01:02:10.645894+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:02:10.645894+0000","last_undegraded":"2024-09-21T01:02:10.645894+0000","last_fullsized":"2024-09-21T01:02:10.645894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317295+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:02:11.317295+0000","last_peered":"2024-09-21T01:02:11.317295+0000","last_clean":"2024-09-21T01:02:11.317295+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:02:11.317295+0000","last_undegraded":"2024-09-21T01:02:11.317295+0000","last_fullsized":"2024-09-21T01:02:11.317295+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73150,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":435,"num_read_kb":296,"num_write":170,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":219450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678055,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1044480,"data_stored":535286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874284,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28732,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716420,"statfs":{"total":95995035648,"available":95965614080,"internally_reserved":0,"allocated":1740800,"data_stored":1195777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037811,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":539596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234040,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":535398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1650688,"data_stored":1131720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1036288,"data_stored":530676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790026,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1654784,"data_stored":1122821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953552,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1048576,"data_stored":535717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1048576,"data_stored":530870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378715,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1134592,"data_stored":601085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542242,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1110016,"data_stored":599344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1060864,"data_stored":532064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:02:51.503 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:02:51.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:51 smithi150 ceph-mon[34945]: pgmap v369: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:51.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:51 smithi150 ceph-mon[34945]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:51.926 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 -- 172.21.15.39:0/3494262345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668109c20 msgr2=0x7f0668110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 --2- 172.21.15.39:0/3494262345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668109c20 0x7f0668110450 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0660009ca0 tx=0x7f066002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 -- 172.21.15.39:0/3494262345 shutdown_connections 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 --2- 172.21.15.39:0/3494262345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668109c20 0x7f0668110450 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 --2- 172.21.15.39:0/3494262345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 0x7f06681096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 --2- 172.21.15.39:0/3494262345 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0668103990 0x7f0668103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 -- 172.21.15.39:0/3494262345 >> 172.21.15.39:0/3494262345 conn(0x7f06680ff440 msgr2=0x7f0668101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:52.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 -- 172.21.15.39:0/3494262345 shutdown_connections 2024-09-21T01:02:52.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.738+0000 7f066e400640 1 -- 172.21.15.39:0/3494262345 wait complete. 2024-09-21T01:02:52.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 Processor -- start 2024-09-21T01:02:52.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 -- start start 2024-09-21T01:02:52.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 0x7f0668075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:52.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 0x7f0668075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:52.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0668109c20 0x7f0668076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f06681138e0 con 0x7f0668103990 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0668113760 con 0x7f06681092d0 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066e400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0668113a60 con 0x7f0668109c20 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.740+0000 7f066d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 0x7f0668075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 0x7f0668075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33608/0 (socket says 172.21.15.39:33608) 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 -- 172.21.15.39:0/2014365864 learned_addr learned my addr 172.21.15.39:0/2014365864 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066da00640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0668109c20 0x7f0668076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f0667e00640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 0x7f0668075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 -- 172.21.15.39:0/2014365864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0668109c20 msgr2=0x7f0668076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0668109c20 0x7f0668076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 -- 172.21.15.39:0/2014365864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 msgr2=0x7f0668075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 0x7f0668075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 -- 172.21.15.39:0/2014365864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06681a8590 con 0x7f0668103990 2024-09-21T01:02:52.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f0667e00640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 0x7f0668075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:02:52.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.741+0000 7f066d000640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 0x7f0668075710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f065c00e9f0 tx=0x7f065c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:52.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.742+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c00cd80 con 0x7f0668103990 2024-09-21T01:02:52.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.742+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f065c0054d0 con 0x7f0668103990 2024-09-21T01:02:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.742+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06681a8880 con 0x7f0668103990 2024-09-21T01:02:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.742+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c010690 con 0x7f0668103990 2024-09-21T01:02:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.742+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06681a8d70 con 0x7f0668103990 2024-09-21T01:02:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.743+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0668103da0 con 0x7f0668103990 2024-09-21T01:02:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.748+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f065c004e30 con 0x7f0668103990 2024-09-21T01:02:52.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.750+0000 7f0665600640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06500b65f0 0x7f06500b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:52.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.750+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f065c0e8090 con 0x7f0668103990 2024-09-21T01:02:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.750+0000 7f0667e00640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06500b65f0 0x7f06500b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.751+0000 7f0667e00640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06500b65f0 0x7f06500b8b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0668102c20 tx=0x7f0654005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.752+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f065c056710 con 0x7f0668103990 2024-09-21T01:02:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:02:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:02:52.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:52 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:02:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:02:52.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:52 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:02:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:02:52.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:52 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:02:52.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.952+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0668063800 con 0x7f06500b65f0 2024-09-21T01:02:52.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.959+0000 7f0665600640 1 -- 172.21.15.39:0/2014365864 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340359 (secure 0 0 0) 0x7f0668063800 con 0x7f06500b65f0 2024-09-21T01:02:52.964 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:52.965 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06500b65f0 msgr2=0x7f06500b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06500b65f0 0x7f06500b8b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0668102c20 tx=0x7f0654005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 msgr2=0x7f0668075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 0x7f0668075710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f065c00e9f0 tx=0x7f065c00edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 shutdown_connections 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0668109c20 0x7f0668076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06500b65f0 0x7f06500b8b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06681092d0 0x7f0668075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 --2- 172.21.15.39:0/2014365864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0668103990 0x7f0668075710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:52.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 >> 172.21.15.39:0/2014365864 conn(0x7f06680ff440 msgr2=0x7f06681068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:52.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 shutdown_connections 2024-09-21T01:02:52.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:52.965+0000 7f066e400640 1 -- 172.21.15.39:0/2014365864 wait complete. 2024-09-21T01:02:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:53 smithi039 ceph-mon[30415]: pgmap v370: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:02:53.796 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-21T01:02:51.922388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663752,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":526,"num_read_kb":365,"num_write":295,"num_write_kb":2277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371936,"kb_used_data":14300,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569888,"statfs":{"total":1151940427776,"available":1151559565312,"internally_reserved":0,"allocated":14643200,"data_stored":8390354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003940"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489181+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:02:10.489181+0000","last_peered":"2024-09-21T01:02:10.489181+0000","last_clean":"2024-09-21T01:02:10.489181+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:02:10.489181+0000","last_undegraded":"2024-09-21T01:02:10.489181+0000","last_fullsized":"2024-09-21T01:02:10.489181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142862+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:02:11.142862+0000","last_peered":"2024-09-21T01:02:11.142862+0000","last_clean":"2024-09-21T01:02:11.142862+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:02:11.142862+0000","last_undegraded":"2024-09-21T01:02:11.142862+0000","last_fullsized":"2024-09-21T01:02:11.142862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485231+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:02:10.485231+0000","last_peered":"2024-09-21T01:02:10.485231+0000","last_clean":"2024-09-21T01:02:10.485231+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:02:10.485231+0000","last_undegraded":"2024-09-21T01:02:10.485231+0000","last_fullsized":"2024-09-21T01:02:10.485231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.144022+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:02:11.144022+0000","last_peered":"2024-09-21T01:02:11.144022+0000","last_clean":"2024-09-21T01:02:11.144022+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:02:11.144022+0000","last_undegraded":"2024-09-21T01:02:11.144022+0000","last_fullsized":"2024-09-21T01:02:11.144022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645212+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:02:10.645212+0000","last_peered":"2024-09-21T01:02:10.645212+0000","last_clean":"2024-09-21T01:02:10.645212+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:02:10.645212+0000","last_undegraded":"2024-09-21T01:02:10.645212+0000","last_fullsized":"2024-09-21T01:02:10.645212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487212+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:02:10.487212+0000","last_peered":"2024-09-21T01:02:10.487212+0000","last_clean":"2024-09-21T01:02:10.487212+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:02:10.487212+0000","last_undegraded":"2024-09-21T01:02:10.487212+0000","last_fullsized":"2024-09-21T01:02:10.487212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137204+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:02:11.137204+0000","last_peered":"2024-09-21T01:02:11.137204+0000","last_clean":"2024-09-21T01:02:11.137204+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:02:11.137204+0000","last_undegraded":"2024-09-21T01:02:11.137204+0000","last_fullsized":"2024-09-21T01:02:11.137204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964115+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:02:10.964115+0000","last_peered":"2024-09-21T01:02:10.964115+0000","last_clean":"2024-09-21T01:02:10.964115+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:02:10.964115+0000","last_undegraded":"2024-09-21T01:02:10.964115+0000","last_fullsized":"2024-09-21T01:02:10.964115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487333+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:02:10.487333+0000","last_peered":"2024-09-21T01:02:10.487333+0000","last_clean":"2024-09-21T01:02:10.487333+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:02:10.487333+0000","last_undegraded":"2024-09-21T01:02:10.487333+0000","last_fullsized":"2024-09-21T01:02:10.487333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136964+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:02:11.136964+0000","last_peered":"2024-09-21T01:02:11.136964+0000","last_clean":"2024-09-21T01:02:11.136964+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:02:11.136964+0000","last_undegraded":"2024-09-21T01:02:11.136964+0000","last_fullsized":"2024-09-21T01:02:11.136964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489438+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:02:10.489438+0000","last_peered":"2024-09-21T01:02:10.489438+0000","last_clean":"2024-09-21T01:02:10.489438+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:02:10.489438+0000","last_undegraded":"2024-09-21T01:02:10.489438+0000","last_fullsized":"2024-09-21T01:02:10.489438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487284+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:02:10.487284+0000","last_peered":"2024-09-21T01:02:10.487284+0000","last_clean":"2024-09-21T01:02:10.487284+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:02:10.487284+0000","last_undegraded":"2024-09-21T01:02:10.487284+0000","last_fullsized":"2024-09-21T01:02:10.487284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490278+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:02:10.490278+0000","last_peered":"2024-09-21T01:02:10.490278+0000","last_clean":"2024-09-21T01:02:10.490278+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:02:10.490278+0000","last_undegraded":"2024-09-21T01:02:10.490278+0000","last_fullsized":"2024-09-21T01:02:10.490278+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136801+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:02:11.136801+0000","last_peered":"2024-09-21T01:02:11.136801+0000","last_clean":"2024-09-21T01:02:11.136801+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:02:11.136801+0000","last_undegraded":"2024-09-21T01:02:11.136801+0000","last_fullsized":"2024-09-21T01:02:11.136801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487975+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:02:10.487975+0000","last_peered":"2024-09-21T01:02:10.487975+0000","last_clean":"2024-09-21T01:02:10.487975+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:02:10.487975+0000","last_undegraded":"2024-09-21T01:02:10.487975+0000","last_fullsized":"2024-09-21T01:02:10.487975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143467+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:02:11.143467+0000","last_peered":"2024-09-21T01:02:11.143467+0000","last_clean":"2024-09-21T01:02:11.143467+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:02:11.143467+0000","last_undegraded":"2024-09-21T01:02:11.143467+0000","last_fullsized":"2024-09-21T01:02:11.143467+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485342+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:02:10.485342+0000","last_peered":"2024-09-21T01:02:10.485342+0000","last_clean":"2024-09-21T01:02:10.485342+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:02:10.485342+0000","last_undegraded":"2024-09-21T01:02:10.485342+0000","last_fullsized":"2024-09-21T01:02:10.485342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487888+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:02:10.487888+0000","last_peered":"2024-09-21T01:02:10.487888+0000","last_clean":"2024-09-21T01:02:10.487888+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:02:10.487888+0000","last_undegraded":"2024-09-21T01:02:10.487888+0000","last_fullsized":"2024-09-21T01:02:10.487888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316462+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:02:11.316462+0000","last_peered":"2024-09-21T01:02:11.316462+0000","last_clean":"2024-09-21T01:02:11.316462+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:02:11.316462+0000","last_undegraded":"2024-09-21T01:02:11.316462+0000","last_fullsized":"2024-09-21T01:02:11.316462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143293+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:02:11.143293+0000","last_peered":"2024-09-21T01:02:11.143293+0000","last_clean":"2024-09-21T01:02:11.143293+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:02:11.143293+0000","last_undegraded":"2024-09-21T01:02:11.143293+0000","last_fullsized":"2024-09-21T01:02:11.143293+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488334+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:02:10.488334+0000","last_peered":"2024-09-21T01:02:10.488334+0000","last_clean":"2024-09-21T01:02:10.488334+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:02:10.488334+0000","last_undegraded":"2024-09-21T01:02:10.488334+0000","last_fullsized":"2024-09-21T01:02:10.488334+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136709+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:02:11.136709+0000","last_peered":"2024-09-21T01:02:11.136709+0000","last_clean":"2024-09-21T01:02:11.136709+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:02:11.136709+0000","last_undegraded":"2024-09-21T01:02:11.136709+0000","last_fullsized":"2024-09-21T01:02:11.136709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964583+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:02:10.964583+0000","last_peered":"2024-09-21T01:02:10.964583+0000","last_clean":"2024-09-21T01:02:10.964583+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:02:10.964583+0000","last_undegraded":"2024-09-21T01:02:10.964583+0000","last_fullsized":"2024-09-21T01:02:10.964583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488036+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:02:10.488036+0000","last_peered":"2024-09-21T01:02:10.488036+0000","last_clean":"2024-09-21T01:02:10.488036+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:02:10.488036+0000","last_undegraded":"2024-09-21T01:02:10.488036+0000","last_fullsized":"2024-09-21T01:02:10.488036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645873+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:02:10.645873+0000","last_peered":"2024-09-21T01:02:10.645873+0000","last_clean":"2024-09-21T01:02:10.645873+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:02:10.645873+0000","last_undegraded":"2024-09-21T01:02:10.645873+0000","last_fullsized":"2024-09-21T01:02:10.645873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488427+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:02:10.488427+0000","last_peered":"2024-09-21T01:02:10.488427+0000","last_clean":"2024-09-21T01:02:10.488427+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:02:10.488427+0000","last_undegraded":"2024-09-21T01:02:10.488427+0000","last_fullsized":"2024-09-21T01:02:10.488427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485574+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:02:10.485574+0000","last_peered":"2024-09-21T01:02:10.485574+0000","last_clean":"2024-09-21T01:02:10.485574+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:02:10.485574+0000","last_undegraded":"2024-09-21T01:02:10.485574+0000","last_fullsized":"2024-09-21T01:02:10.485574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142902+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:02:11.142902+0000","last_peered":"2024-09-21T01:02:11.142902+0000","last_clean":"2024-09-21T01:02:11.142902+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:02:11.142902+0000","last_undegraded":"2024-09-21T01:02:11.142902+0000","last_fullsized":"2024-09-21T01:02:11.142902+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488972+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:02:10.488972+0000","last_peered":"2024-09-21T01:02:10.488972+0000","last_clean":"2024-09-21T01:02:10.488972+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:02:10.488972+0000","last_undegraded":"2024-09-21T01:02:10.488972+0000","last_fullsized":"2024-09-21T01:02:10.488972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142917+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:02:11.142917+0000","last_peered":"2024-09-21T01:02:11.142917+0000","last_clean":"2024-09-21T01:02:11.142917+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:02:11.142917+0000","last_undegraded":"2024-09-21T01:02:11.142917+0000","last_fullsized":"2024-09-21T01:02:11.142917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484965+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:02:10.484965+0000","last_peered":"2024-09-21T01:02:10.484965+0000","last_clean":"2024-09-21T01:02:10.484965+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:02:10.484965+0000","last_undegraded":"2024-09-21T01:02:10.484965+0000","last_fullsized":"2024-09-21T01:02:10.484965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137262+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:02:11.137262+0000","last_peered":"2024-09-21T01:02:11.137262+0000","last_clean":"2024-09-21T01:02:11.137262+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:02:11.137262+0000","last_undegraded":"2024-09-21T01:02:11.137262+0000","last_fullsized":"2024-09-21T01:02:11.137262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316884+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:02:11.316884+0000","last_peered":"2024-09-21T01:02:11.316884+0000","last_clean":"2024-09-21T01:02:11.316884+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:02:11.316884+0000","last_undegraded":"2024-09-21T01:02:11.316884+0000","last_fullsized":"2024-09-21T01:02:11.316884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517184+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:02:10.517184+0000","last_peered":"2024-09-21T01:02:10.517184+0000","last_clean":"2024-09-21T01:02:10.517184+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:02:10.517184+0000","last_undegraded":"2024-09-21T01:02:10.517184+0000","last_fullsized":"2024-09-21T01:02:10.517184+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488662+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:02:10.488662+0000","last_peered":"2024-09-21T01:02:10.488662+0000","last_clean":"2024-09-21T01:02:10.488662+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:02:10.488662+0000","last_undegraded":"2024-09-21T01:02:10.488662+0000","last_fullsized":"2024-09-21T01:02:10.488662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136488+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:02:11.136488+0000","last_peered":"2024-09-21T01:02:11.136488+0000","last_clean":"2024-09-21T01:02:11.136488+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:02:11.136488+0000","last_undegraded":"2024-09-21T01:02:11.136488+0000","last_fullsized":"2024-09-21T01:02:11.136488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517079+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:02:10.517079+0000","last_peered":"2024-09-21T01:02:10.517079+0000","last_clean":"2024-09-21T01:02:10.517079+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:02:10.517079+0000","last_undegraded":"2024-09-21T01:02:10.517079+0000","last_fullsized":"2024-09-21T01:02:10.517079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490887+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:02:10.490887+0000","last_peered":"2024-09-21T01:02:10.490887+0000","last_clean":"2024-09-21T01:02:10.490887+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:02:10.490887+0000","last_undegraded":"2024-09-21T01:02:10.490887+0000","last_fullsized":"2024-09-21T01:02:10.490887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484758+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:02:10.484758+0000","last_peered":"2024-09-21T01:02:10.484758+0000","last_clean":"2024-09-21T01:02:10.484758+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:02:10.484758+0000","last_undegraded":"2024-09-21T01:02:10.484758+0000","last_fullsized":"2024-09-21T01:02:10.484758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489249+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:02:10.489249+0000","last_peered":"2024-09-21T01:02:10.489249+0000","last_clean":"2024-09-21T01:02:10.489249+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:02:10.489249+0000","last_undegraded":"2024-09-21T01:02:10.489249+0000","last_fullsized":"2024-09-21T01:02:10.489249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:46.491081+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:02:46.491081+0000","last_peered":"2024-09-21T01:02:46.491081+0000","last_clean":"2024-09-21T01:02:46.491081+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:02:46.491081+0000","last_undegraded":"2024-09-21T01:02:46.491081+0000","last_fullsized":"2024-09-21T01:02:46.491081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485032+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:02:10.485032+0000","last_peered":"2024-09-21T01:02:10.485032+0000","last_clean":"2024-09-21T01:02:10.485032+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:02:10.485032+0000","last_undegraded":"2024-09-21T01:02:10.485032+0000","last_fullsized":"2024-09-21T01:02:10.485032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484255+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:02:10.484255+0000","last_peered":"2024-09-21T01:02:10.484255+0000","last_clean":"2024-09-21T01:02:10.484255+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:02:10.484255+0000","last_undegraded":"2024-09-21T01:02:10.484255+0000","last_fullsized":"2024-09-21T01:02:10.484255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490838+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:02:10.490838+0000","last_peered":"2024-09-21T01:02:10.490838+0000","last_clean":"2024-09-21T01:02:10.490838+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:02:10.490838+0000","last_undegraded":"2024-09-21T01:02:10.490838+0000","last_fullsized":"2024-09-21T01:02:10.490838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489919+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:02:10.489919+0000","last_peered":"2024-09-21T01:02:10.489919+0000","last_clean":"2024-09-21T01:02:10.489919+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:02:10.489919+0000","last_undegraded":"2024-09-21T01:02:10.489919+0000","last_fullsized":"2024-09-21T01:02:10.489919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143430+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:02:11.143430+0000","last_peered":"2024-09-21T01:02:11.143430+0000","last_clean":"2024-09-21T01:02:11.143430+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:02:11.143430+0000","last_undegraded":"2024-09-21T01:02:11.143430+0000","last_fullsized":"2024-09-21T01:02:11.143430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487931+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:02:10.487931+0000","last_peered":"2024-09-21T01:02:10.487931+0000","last_clean":"2024-09-21T01:02:10.487931+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:02:10.487931+0000","last_undegraded":"2024-09-21T01:02:10.487931+0000","last_fullsized":"2024-09-21T01:02:10.487931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490885+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:02:10.490885+0000","last_peered":"2024-09-21T01:02:10.490885+0000","last_clean":"2024-09-21T01:02:10.490885+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:02:10.490885+0000","last_undegraded":"2024-09-21T01:02:10.490885+0000","last_fullsized":"2024-09-21T01:02:10.490885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490147+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:02:10.490147+0000","last_peered":"2024-09-21T01:02:10.490147+0000","last_clean":"2024-09-21T01:02:10.490147+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:02:10.490147+0000","last_undegraded":"2024-09-21T01:02:10.490147+0000","last_fullsized":"2024-09-21T01:02:10.490147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490730+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:02:10.490730+0000","last_peered":"2024-09-21T01:02:10.490730+0000","last_clean":"2024-09-21T01:02:10.490730+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:02:10.490730+0000","last_undegraded":"2024-09-21T01:02:10.490730+0000","last_fullsized":"2024-09-21T01:02:10.490730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485884+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:02:10.485884+0000","last_peered":"2024-09-21T01:02:10.485884+0000","last_clean":"2024-09-21T01:02:10.485884+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:02:10.485884+0000","last_undegraded":"2024-09-21T01:02:10.485884+0000","last_fullsized":"2024-09-21T01:02:10.485884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964535+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:02:10.964535+0000","last_peered":"2024-09-21T01:02:10.964535+0000","last_clean":"2024-09-21T01:02:10.964535+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:02:10.964535+0000","last_undegraded":"2024-09-21T01:02:10.964535+0000","last_fullsized":"2024-09-21T01:02:10.964535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490480+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:02:10.490480+0000","last_peered":"2024-09-21T01:02:10.490480+0000","last_clean":"2024-09-21T01:02:10.490480+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:02:10.490480+0000","last_undegraded":"2024-09-21T01:02:10.490480+0000","last_fullsized":"2024-09-21T01:02:10.490480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489782+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:02:10.489782+0000","last_peered":"2024-09-21T01:02:10.489782+0000","last_clean":"2024-09-21T01:02:10.489782+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:02:10.489782+0000","last_undegraded":"2024-09-21T01:02:10.489782+0000","last_fullsized":"2024-09-21T01:02:10.489782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645520+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:02:10.645520+0000","last_peered":"2024-09-21T01:02:10.645520+0000","last_clean":"2024-09-21T01:02:10.645520+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:02:10.645520+0000","last_undegraded":"2024-09-21T01:02:10.645520+0000","last_fullsized":"2024-09-21T01:02:10.645520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490362+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:02:10.490362+0000","last_peered":"2024-09-21T01:02:10.490362+0000","last_clean":"2024-09-21T01:02:10.490362+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:02:10.490362+0000","last_undegraded":"2024-09-21T01:02:10.490362+0000","last_fullsized":"2024-09-21T01:02:10.490362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485031+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:02:10.485031+0000","last_peered":"2024-09-21T01:02:10.485031+0000","last_clean":"2024-09-21T01:02:10.485031+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:02:10.485031+0000","last_undegraded":"2024-09-21T01:02:10.485031+0000","last_fullsized":"2024-09-21T01:02:10.485031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488992+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:02:10.488992+0000","last_peered":"2024-09-21T01:02:10.488992+0000","last_clean":"2024-09-21T01:02:10.488992+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:02:10.488992+0000","last_undegraded":"2024-09-21T01:02:10.488992+0000","last_fullsized":"2024-09-21T01:02:10.488992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316507+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:02:11.316507+0000","last_peered":"2024-09-21T01:02:11.316507+0000","last_clean":"2024-09-21T01:02:11.316507+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:02:11.316507+0000","last_undegraded":"2024-09-21T01:02:11.316507+0000","last_fullsized":"2024-09-21T01:02:11.316507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317248+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:02:11.317248+0000","last_peered":"2024-09-21T01:02:11.317248+0000","last_clean":"2024-09-21T01:02:11.317248+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:02:11.317248+0000","last_undegraded":"2024-09-21T01:02:11.317248+0000","last_fullsized":"2024-09-21T01:02:11.317248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489724+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:02:10.489724+0000","last_peered":"2024-09-21T01:02:10.489724+0000","last_clean":"2024-09-21T01:02:10.489724+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:02:10.489724+0000","last_undegraded":"2024-09-21T01:02:10.489724+0000","last_fullsized":"2024-09-21T01:02:10.489724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.966099+0000","last_change":"2024-09-21T01:02:10.966004+0000","last_active":"2024-09-21T01:02:10.966099+0000","last_peered":"2024-09-21T01:02:10.966099+0000","last_clean":"2024-09-21T01:02:10.966099+0000","last_became_active":"2024-09-21T01:02:10.965645+0000","last_became_peered":"2024-09-21T01:02:10.965645+0000","last_unstale":"2024-09-21T01:02:10.966099+0000","last_undegraded":"2024-09-21T01:02:10.966099+0000","last_fullsized":"2024-09-21T01:02:10.966099+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:12.070017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136545+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:02:11.136545+0000","last_peered":"2024-09-21T01:02:11.136545+0000","last_clean":"2024-09-21T01:02:11.136545+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:02:11.136545+0000","last_undegraded":"2024-09-21T01:02:11.136545+0000","last_fullsized":"2024-09-21T01:02:11.136545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484951+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:02:10.484951+0000","last_peered":"2024-09-21T01:02:10.484951+0000","last_clean":"2024-09-21T01:02:10.484951+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:02:10.484951+0000","last_undegraded":"2024-09-21T01:02:10.484951+0000","last_fullsized":"2024-09-21T01:02:10.484951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489976+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:02:10.489976+0000","last_peered":"2024-09-21T01:02:10.489976+0000","last_clean":"2024-09-21T01:02:10.489976+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:02:10.489976+0000","last_undegraded":"2024-09-21T01:02:10.489976+0000","last_fullsized":"2024-09-21T01:02:10.489976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:50.499439+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:02:50.499439+0000","last_peered":"2024-09-21T01:02:50.499439+0000","last_clean":"2024-09-21T01:02:50.499439+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:02:50.499439+0000","last_undegraded":"2024-09-21T01:02:50.499439+0000","last_fullsized":"2024-09-21T01:02:50.499439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489100+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:02:10.489100+0000","last_peered":"2024-09-21T01:02:10.489100+0000","last_clean":"2024-09-21T01:02:10.489100+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:02:10.489100+0000","last_undegraded":"2024-09-21T01:02:10.489100+0000","last_fullsized":"2024-09-21T01:02:10.489100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490595+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:02:10.490595+0000","last_peered":"2024-09-21T01:02:10.490595+0000","last_clean":"2024-09-21T01:02:10.490595+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:02:10.490595+0000","last_undegraded":"2024-09-21T01:02:10.490595+0000","last_fullsized":"2024-09-21T01:02:10.490595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489394+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:02:10.489394+0000","last_peered":"2024-09-21T01:02:10.489394+0000","last_clean":"2024-09-21T01:02:10.489394+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:02:10.489394+0000","last_undegraded":"2024-09-21T01:02:10.489394+0000","last_fullsized":"2024-09-21T01:02:10.489394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316850+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:02:11.316850+0000","last_peered":"2024-09-21T01:02:11.316850+0000","last_clean":"2024-09-21T01:02:11.316850+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:02:11.316850+0000","last_undegraded":"2024-09-21T01:02:11.316850+0000","last_fullsized":"2024-09-21T01:02:11.316850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484391+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:02:10.484391+0000","last_peered":"2024-09-21T01:02:10.484391+0000","last_clean":"2024-09-21T01:02:10.484391+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:02:10.484391+0000","last_undegraded":"2024-09-21T01:02:10.484391+0000","last_fullsized":"2024-09-21T01:02:10.484391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485499+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:02:10.485499+0000","last_peered":"2024-09-21T01:02:10.485499+0000","last_clean":"2024-09-21T01:02:10.485499+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:02:10.485499+0000","last_undegraded":"2024-09-21T01:02:10.485499+0000","last_fullsized":"2024-09-21T01:02:10.485499+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489623+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:02:10.489623+0000","last_peered":"2024-09-21T01:02:10.489623+0000","last_clean":"2024-09-21T01:02:10.489623+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:02:10.489623+0000","last_undegraded":"2024-09-21T01:02:10.489623+0000","last_fullsized":"2024-09-21T01:02:10.489623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.135886+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:02:11.135886+0000","last_peered":"2024-09-21T01:02:11.135886+0000","last_clean":"2024-09-21T01:02:11.135886+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:02:11.135886+0000","last_undegraded":"2024-09-21T01:02:11.135886+0000","last_fullsized":"2024-09-21T01:02:11.135886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":227,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:49.485433+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:02:49.485433+0000","last_peered":"2024-09-21T01:02:49.485433+0000","last_clean":"2024-09-21T01:02:49.485433+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:02:49.485433+0000","last_undegraded":"2024-09-21T01:02:49.485433+0000","last_fullsized":"2024-09-21T01:02:49.485433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":148,"num_read_kb":141,"num_write":36,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488137+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:02:10.488137+0000","last_peered":"2024-09-21T01:02:10.488137+0000","last_clean":"2024-09-21T01:02:10.488137+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:02:10.488137+0000","last_undegraded":"2024-09-21T01:02:10.488137+0000","last_fullsized":"2024-09-21T01:02:10.488137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964114+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:02:10.964114+0000","last_peered":"2024-09-21T01:02:10.964114+0000","last_clean":"2024-09-21T01:02:10.964114+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:02:10.964114+0000","last_undegraded":"2024-09-21T01:02:10.964114+0000","last_fullsized":"2024-09-21T01:02:10.964114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489715+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:02:10.489715+0000","last_peered":"2024-09-21T01:02:10.489715+0000","last_clean":"2024-09-21T01:02:10.489715+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:02:10.489715+0000","last_undegraded":"2024-09-21T01:02:10.489715+0000","last_fullsized":"2024-09-21T01:02:10.489715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487635+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:02:10.487635+0000","last_peered":"2024-09-21T01:02:10.487635+0000","last_clean":"2024-09-21T01:02:10.487635+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:02:10.487635+0000","last_undegraded":"2024-09-21T01:02:10.487635+0000","last_fullsized":"2024-09-21T01:02:10.487635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.644982+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:02:10.644982+0000","last_peered":"2024-09-21T01:02:10.644982+0000","last_clean":"2024-09-21T01:02:10.644982+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:02:10.644982+0000","last_undegraded":"2024-09-21T01:02:10.644982+0000","last_fullsized":"2024-09-21T01:02:10.644982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490215+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:02:10.490215+0000","last_peered":"2024-09-21T01:02:10.490215+0000","last_clean":"2024-09-21T01:02:10.490215+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:02:10.490215+0000","last_undegraded":"2024-09-21T01:02:10.490215+0000","last_fullsized":"2024-09-21T01:02:10.490215+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487904+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:02:10.487904+0000","last_peered":"2024-09-21T01:02:10.487904+0000","last_clean":"2024-09-21T01:02:10.487904+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:02:10.487904+0000","last_undegraded":"2024-09-21T01:02:10.487904+0000","last_fullsized":"2024-09-21T01:02:10.487904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964130+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:02:10.964130+0000","last_peered":"2024-09-21T01:02:10.964130+0000","last_clean":"2024-09-21T01:02:10.964130+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:02:10.964130+0000","last_undegraded":"2024-09-21T01:02:10.964130+0000","last_fullsized":"2024-09-21T01:02:10.964130+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:48.483835+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:02:48.483835+0000","last_peered":"2024-09-21T01:02:48.483835+0000","last_clean":"2024-09-21T01:02:48.483835+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:02:48.483835+0000","last_undegraded":"2024-09-21T01:02:48.483835+0000","last_fullsized":"2024-09-21T01:02:48.483835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143060+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:02:11.143060+0000","last_peered":"2024-09-21T01:02:11.143060+0000","last_clean":"2024-09-21T01:02:11.143060+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:02:11.143060+0000","last_undegraded":"2024-09-21T01:02:11.143060+0000","last_fullsized":"2024-09-21T01:02:11.143060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645694+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:02:10.645694+0000","last_peered":"2024-09-21T01:02:10.645694+0000","last_clean":"2024-09-21T01:02:10.645694+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:02:10.645694+0000","last_undegraded":"2024-09-21T01:02:10.645694+0000","last_fullsized":"2024-09-21T01:02:10.645694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488122+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:02:10.488122+0000","last_peered":"2024-09-21T01:02:10.488122+0000","last_clean":"2024-09-21T01:02:10.488122+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:02:10.488122+0000","last_undegraded":"2024-09-21T01:02:10.488122+0000","last_fullsized":"2024-09-21T01:02:10.488122+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489363+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:02:10.489363+0000","last_peered":"2024-09-21T01:02:10.489363+0000","last_clean":"2024-09-21T01:02:10.489363+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:02:10.489363+0000","last_undegraded":"2024-09-21T01:02:10.489363+0000","last_fullsized":"2024-09-21T01:02:10.489363+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317103+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:02:11.317103+0000","last_peered":"2024-09-21T01:02:11.317103+0000","last_clean":"2024-09-21T01:02:11.317103+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:02:11.317103+0000","last_undegraded":"2024-09-21T01:02:11.317103+0000","last_fullsized":"2024-09-21T01:02:11.317103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142654+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:02:11.142654+0000","last_peered":"2024-09-21T01:02:11.142654+0000","last_clean":"2024-09-21T01:02:11.142654+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:02:11.142654+0000","last_undegraded":"2024-09-21T01:02:11.142654+0000","last_fullsized":"2024-09-21T01:02:11.142654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484193+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:02:10.484193+0000","last_peered":"2024-09-21T01:02:10.484193+0000","last_clean":"2024-09-21T01:02:10.484193+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:02:10.484193+0000","last_undegraded":"2024-09-21T01:02:10.484193+0000","last_fullsized":"2024-09-21T01:02:10.484193+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517512+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:02:10.517512+0000","last_peered":"2024-09-21T01:02:10.517512+0000","last_clean":"2024-09-21T01:02:10.517512+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:02:10.517512+0000","last_undegraded":"2024-09-21T01:02:10.517512+0000","last_fullsized":"2024-09-21T01:02:10.517512+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489999+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:02:10.489999+0000","last_peered":"2024-09-21T01:02:10.489999+0000","last_clean":"2024-09-21T01:02:10.489999+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:02:10.489999+0000","last_undegraded":"2024-09-21T01:02:10.489999+0000","last_fullsized":"2024-09-21T01:02:10.489999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485566+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:02:10.485566+0000","last_peered":"2024-09-21T01:02:10.485566+0000","last_clean":"2024-09-21T01:02:10.485566+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:02:10.485566+0000","last_undegraded":"2024-09-21T01:02:10.485566+0000","last_fullsized":"2024-09-21T01:02:10.485566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":124,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:50.487596+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:02:50.487596+0000","last_peered":"2024-09-21T01:02:50.487596+0000","last_clean":"2024-09-21T01:02:50.487596+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:02:50.487596+0000","last_undegraded":"2024-09-21T01:02:50.487596+0000","last_fullsized":"2024-09-21T01:02:50.487596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.135852+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:02:11.135852+0000","last_peered":"2024-09-21T01:02:11.135852+0000","last_clean":"2024-09-21T01:02:11.135852+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:02:11.135852+0000","last_undegraded":"2024-09-21T01:02:11.135852+0000","last_fullsized":"2024-09-21T01:02:11.135852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485531+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:02:10.485531+0000","last_peered":"2024-09-21T01:02:10.485531+0000","last_clean":"2024-09-21T01:02:10.485531+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:02:10.485531+0000","last_undegraded":"2024-09-21T01:02:10.485531+0000","last_fullsized":"2024-09-21T01:02:10.485531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316629+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:02:11.316629+0000","last_peered":"2024-09-21T01:02:11.316629+0000","last_clean":"2024-09-21T01:02:11.316629+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:02:11.316629+0000","last_undegraded":"2024-09-21T01:02:11.316629+0000","last_fullsized":"2024-09-21T01:02:11.316629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485868+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:02:10.485868+0000","last_peered":"2024-09-21T01:02:10.485868+0000","last_clean":"2024-09-21T01:02:10.485868+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:02:10.485868+0000","last_undegraded":"2024-09-21T01:02:10.485868+0000","last_fullsized":"2024-09-21T01:02:10.485868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485364+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:02:10.485364+0000","last_peered":"2024-09-21T01:02:10.485364+0000","last_clean":"2024-09-21T01:02:10.485364+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:02:10.485364+0000","last_undegraded":"2024-09-21T01:02:10.485364+0000","last_fullsized":"2024-09-21T01:02:10.485364+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645189+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:02:10.645189+0000","last_peered":"2024-09-21T01:02:10.645189+0000","last_clean":"2024-09-21T01:02:10.645189+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:02:10.645189+0000","last_undegraded":"2024-09-21T01:02:10.645189+0000","last_fullsized":"2024-09-21T01:02:10.645189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.647189+0000","last_change":"2024-09-21T01:02:10.647066+0000","last_active":"2024-09-21T01:02:10.647189+0000","last_peered":"2024-09-21T01:02:10.647189+0000","last_clean":"2024-09-21T01:02:10.647189+0000","last_became_active":"2024-09-21T01:02:10.646487+0000","last_became_peered":"2024-09-21T01:02:10.646487+0000","last_unstale":"2024-09-21T01:02:10.647189+0000","last_undegraded":"2024-09-21T01:02:10.647189+0000","last_fullsized":"2024-09-21T01:02:10.647189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:19:28.445094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316477+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:02:11.316477+0000","last_peered":"2024-09-21T01:02:11.316477+0000","last_clean":"2024-09-21T01:02:11.316477+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:02:11.316477+0000","last_undegraded":"2024-09-21T01:02:11.316477+0000","last_fullsized":"2024-09-21T01:02:11.316477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485777+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:02:10.485777+0000","last_peered":"2024-09-21T01:02:10.485777+0000","last_clean":"2024-09-21T01:02:10.485777+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:02:10.485777+0000","last_undegraded":"2024-09-21T01:02:10.485777+0000","last_fullsized":"2024-09-21T01:02:10.485777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142910+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:02:11.142910+0000","last_peered":"2024-09-21T01:02:11.142910+0000","last_clean":"2024-09-21T01:02:11.142910+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:02:11.142910+0000","last_undegraded":"2024-09-21T01:02:11.142910+0000","last_fullsized":"2024-09-21T01:02:11.142910+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489735+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:02:10.489735+0000","last_peered":"2024-09-21T01:02:10.489735+0000","last_clean":"2024-09-21T01:02:10.489735+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:02:10.489735+0000","last_undegraded":"2024-09-21T01:02:10.489735+0000","last_fullsized":"2024-09-21T01:02:10.489735+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.318874+0000","last_change":"2024-09-21T01:02:11.318295+0000","last_active":"2024-09-21T01:02:11.318874+0000","last_peered":"2024-09-21T01:02:11.318874+0000","last_clean":"2024-09-21T01:02:11.318874+0000","last_became_active":"2024-09-21T01:02:11.317912+0000","last_became_peered":"2024-09-21T01:02:11.317912+0000","last_unstale":"2024-09-21T01:02:11.318874+0000","last_undegraded":"2024-09-21T01:02:11.318874+0000","last_fullsized":"2024-09-21T01:02:11.318874+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:53:47.063340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964148+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:02:10.964148+0000","last_peered":"2024-09-21T01:02:10.964148+0000","last_clean":"2024-09-21T01:02:10.964148+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:02:10.964148+0000","last_undegraded":"2024-09-21T01:02:10.964148+0000","last_fullsized":"2024-09-21T01:02:10.964148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484627+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:02:10.484627+0000","last_peered":"2024-09-21T01:02:10.484627+0000","last_clean":"2024-09-21T01:02:10.484627+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:02:10.484627+0000","last_undegraded":"2024-09-21T01:02:10.484627+0000","last_fullsized":"2024-09-21T01:02:10.484627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316477+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:02:11.316477+0000","last_peered":"2024-09-21T01:02:11.316477+0000","last_clean":"2024-09-21T01:02:11.316477+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:02:11.316477+0000","last_undegraded":"2024-09-21T01:02:11.316477+0000","last_fullsized":"2024-09-21T01:02:11.316477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485672+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:02:10.485672+0000","last_peered":"2024-09-21T01:02:10.485672+0000","last_clean":"2024-09-21T01:02:10.485672+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:02:10.485672+0000","last_undegraded":"2024-09-21T01:02:10.485672+0000","last_fullsized":"2024-09-21T01:02:10.485672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488903+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:02:10.488903+0000","last_peered":"2024-09-21T01:02:10.488903+0000","last_clean":"2024-09-21T01:02:10.488903+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:02:10.488903+0000","last_undegraded":"2024-09-21T01:02:10.488903+0000","last_fullsized":"2024-09-21T01:02:10.488903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317334+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:02:11.317334+0000","last_peered":"2024-09-21T01:02:11.317334+0000","last_clean":"2024-09-21T01:02:11.317334+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:02:11.317334+0000","last_undegraded":"2024-09-21T01:02:11.317334+0000","last_fullsized":"2024-09-21T01:02:11.317334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964029+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:02:10.964029+0000","last_peered":"2024-09-21T01:02:10.964029+0000","last_clean":"2024-09-21T01:02:10.964029+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:02:10.964029+0000","last_undegraded":"2024-09-21T01:02:10.964029+0000","last_fullsized":"2024-09-21T01:02:10.964029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.644922+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:02:10.644922+0000","last_peered":"2024-09-21T01:02:10.644922+0000","last_clean":"2024-09-21T01:02:10.644922+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:02:10.644922+0000","last_undegraded":"2024-09-21T01:02:10.644922+0000","last_fullsized":"2024-09-21T01:02:10.644922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645391+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:02:10.645391+0000","last_peered":"2024-09-21T01:02:10.645391+0000","last_clean":"2024-09-21T01:02:10.645391+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:02:10.645391+0000","last_undegraded":"2024-09-21T01:02:10.645391+0000","last_fullsized":"2024-09-21T01:02:10.645391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484591+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:02:10.484591+0000","last_peered":"2024-09-21T01:02:10.484591+0000","last_clean":"2024-09-21T01:02:10.484591+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:02:10.484591+0000","last_undegraded":"2024-09-21T01:02:10.484591+0000","last_fullsized":"2024-09-21T01:02:10.484591+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488141+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:02:10.488141+0000","last_peered":"2024-09-21T01:02:10.488141+0000","last_clean":"2024-09-21T01:02:10.488141+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:02:10.488141+0000","last_undegraded":"2024-09-21T01:02:10.488141+0000","last_fullsized":"2024-09-21T01:02:10.488141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490320+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:02:10.490320+0000","last_peered":"2024-09-21T01:02:10.490320+0000","last_clean":"2024-09-21T01:02:10.490320+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:02:10.490320+0000","last_undegraded":"2024-09-21T01:02:10.490320+0000","last_fullsized":"2024-09-21T01:02:10.490320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517008+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:02:10.517008+0000","last_peered":"2024-09-21T01:02:10.517008+0000","last_clean":"2024-09-21T01:02:10.517008+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:02:10.517008+0000","last_undegraded":"2024-09-21T01:02:10.517008+0000","last_fullsized":"2024-09-21T01:02:10.517008+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964290+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:02:10.964290+0000","last_peered":"2024-09-21T01:02:10.964290+0000","last_clean":"2024-09-21T01:02:10.964290+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:02:10.964290+0000","last_undegraded":"2024-09-21T01:02:10.964290+0000","last_fullsized":"2024-09-21T01:02:10.964290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142650+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:02:11.142650+0000","last_peered":"2024-09-21T01:02:11.142650+0000","last_clean":"2024-09-21T01:02:11.142650+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:02:11.142650+0000","last_undegraded":"2024-09-21T01:02:11.142650+0000","last_fullsized":"2024-09-21T01:02:11.142650+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.138407+0000","last_change":"2024-09-21T01:02:11.138284+0000","last_active":"2024-09-21T01:02:11.138407+0000","last_peered":"2024-09-21T01:02:11.138407+0000","last_clean":"2024-09-21T01:02:11.138407+0000","last_became_active":"2024-09-21T01:02:11.137782+0000","last_became_peered":"2024-09-21T01:02:11.137782+0000","last_unstale":"2024-09-21T01:02:11.138407+0000","last_undegraded":"2024-09-21T01:02:11.138407+0000","last_fullsized":"2024-09-21T01:02:11.138407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:57.753015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.516933+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:02:10.516933+0000","last_peered":"2024-09-21T01:02:10.516933+0000","last_clean":"2024-09-21T01:02:10.516933+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:02:10.516933+0000","last_undegraded":"2024-09-21T01:02:10.516933+0000","last_fullsized":"2024-09-21T01:02:10.516933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.135836+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:02:11.135836+0000","last_peered":"2024-09-21T01:02:11.135836+0000","last_clean":"2024-09-21T01:02:11.135836+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:02:11.135836+0000","last_undegraded":"2024-09-21T01:02:11.135836+0000","last_fullsized":"2024-09-21T01:02:11.135836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488426+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:02:10.488426+0000","last_peered":"2024-09-21T01:02:10.488426+0000","last_clean":"2024-09-21T01:02:10.488426+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:02:10.488426+0000","last_undegraded":"2024-09-21T01:02:10.488426+0000","last_fullsized":"2024-09-21T01:02:10.488426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645067+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:02:10.645067+0000","last_peered":"2024-09-21T01:02:10.645067+0000","last_clean":"2024-09-21T01:02:10.645067+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:02:10.645067+0000","last_undegraded":"2024-09-21T01:02:10.645067+0000","last_fullsized":"2024-09-21T01:02:10.645067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645894+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:02:10.645894+0000","last_peered":"2024-09-21T01:02:10.645894+0000","last_clean":"2024-09-21T01:02:10.645894+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:02:10.645894+0000","last_undegraded":"2024-09-21T01:02:10.645894+0000","last_fullsized":"2024-09-21T01:02:10.645894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317330+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:02:11.317330+0000","last_peered":"2024-09-21T01:02:11.317330+0000","last_clean":"2024-09-21T01:02:11.317330+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:02:11.317330+0000","last_undegraded":"2024-09-21T01:02:11.317330+0000","last_fullsized":"2024-09-21T01:02:11.317330+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73150,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":445,"num_read_kb":296,"num_write":170,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":219450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678056,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1044480,"data_stored":535286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874284,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28732,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716420,"statfs":{"total":95995035648,"available":95965614080,"internally_reserved":0,"allocated":1740800,"data_stored":1195777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037811,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":539596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":535398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1650688,"data_stored":1131720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1036288,"data_stored":530676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790027,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1654784,"data_stored":1122821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953553,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1048576,"data_stored":535717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1048576,"data_stored":530870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378715,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1134592,"data_stored":601085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542242,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1110016,"data_stored":599344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1060864,"data_stored":532064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:02:53.800 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T01:02:53.801 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T01:02:53.801 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T01:02:53.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-21T01:02:53.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:53 smithi160 ceph-mon[34696]: pgmap v370: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:02:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:53 smithi150 ceph-mon[34945]: pgmap v370: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:02:54.227 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:54.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:54 smithi039 ceph-mon[30415]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:54.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:54 smithi160 ceph-mon[34696]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:54.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 -- 172.21.15.39:0/1602173327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e981094a0 msgr2=0x7f2e981098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:54.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 --2- 172.21.15.39:0/1602173327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e981094a0 0x7f2e981098b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009900 tx=0x7f2e8402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 -- 172.21.15.39:0/1602173327 shutdown_connections 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 --2- 172.21.15.39:0/1602173327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98109df0 0x7f2e98110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 --2- 172.21.15.39:0/1602173327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e981094a0 0x7f2e981098b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 --2- 172.21.15.39:0/1602173327 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2e98103b60 0x7f2e98103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 -- 172.21.15.39:0/1602173327 >> 172.21.15.39:0/1602173327 conn(0x7f2e9807b850 msgr2=0x7f2e9807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 -- 172.21.15.39:0/1602173327 shutdown_connections 2024-09-21T01:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.882+0000 7f2e9f800640 1 -- 172.21.15.39:0/1602173327 wait complete. 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.883+0000 7f2e9f800640 1 Processor -- start 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.883+0000 7f2e9f800640 1 -- start start 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.883+0000 7f2e9f800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 0x7f2e9819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.883+0000 7f2e9f800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2e981094a0 0x7f2e9819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.883+0000 7f2e9f800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e98109df0 0x7f2e981a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9f800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98113b00 con 0x7f2e98103b60 2024-09-21T01:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9f800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98113980 con 0x7f2e98109df0 2024-09-21T01:02:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9f800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98113c80 con 0x7f2e981094a0 2024-09-21T01:02:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 0x7f2e9819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 0x7f2e9819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43140/0 (socket says 172.21.15.39:43140) 2024-09-21T01:02:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 -- 172.21.15.39:0/294751034 learned_addr learned my addr 172.21.15.39:0/294751034 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e97e00640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2e981094a0 0x7f2e9819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9d800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e98109df0 0x7f2e981a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 -- 172.21.15.39:0/294751034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2e981094a0 msgr2=0x7f2e9819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2e981094a0 0x7f2e9819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 -- 172.21.15.39:0/294751034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e98109df0 msgr2=0x7f2e981a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e98109df0 0x7f2e981a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9ce00640 1 -- 172.21.15.39:0/294751034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e981a2600 con 0x7f2e98103b60 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.884+0000 7f2e9d800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e98109df0 0x7f2e981a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.885+0000 7f2e9ce00640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 0x7f2e9819c690 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c00e9f0 tx=0x7f2e8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.885+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00cd80 con 0x7f2e98103b60 2024-09-21T01:02:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.885+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e8c0054d0 con 0x7f2e98103b60 2024-09-21T01:02:54.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.885+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e981a28f0 con 0x7f2e98103b60 2024-09-21T01:02:54.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.885+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c010690 con 0x7f2e98103b60 2024-09-21T01:02:54.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.885+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e981a9050 con 0x7f2e98103b60 2024-09-21T01:02:54.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.886+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e98103f70 con 0x7f2e98103b60 2024-09-21T01:02:54.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.889+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2e8c004e30 con 0x7f2e98103b60 2024-09-21T01:02:54.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.890+0000 7f2e95600640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2e800b65f0 0x7f2e800b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:54.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.890+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2e8c0e7fb0 con 0x7f2e98103b60 2024-09-21T01:02:54.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.890+0000 7f2e97e00640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2e800b65f0 0x7f2e800b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:54.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.891+0000 7f2e97e00640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2e800b65f0 0x7f2e800b8b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2e8402a9e0 tx=0x7f2e84005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:54.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:54.892+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2e8c0ae1e0 con 0x7f2e98103b60 2024-09-21T01:02:55.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:54 smithi150 ceph-mon[34945]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:02:55.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.132+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2e9819d7c0 con 0x7f2e98103b60 2024-09-21T01:02:55.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.132+0000 7f2e95600640 1 -- 172.21.15.39:0/294751034 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f2e9819d7c0 con 0x7f2e98103b60 2024-09-21T01:02:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:55.134 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2e800b65f0 msgr2=0x7f2e800b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2e800b65f0 0x7f2e800b8b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2e8402a9e0 tx=0x7f2e84005a20 comp rx=0 tx=0).stop 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 msgr2=0x7f2e9819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 0x7f2e9819c690 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c00e9f0 tx=0x7f2e8c00edf0 comp rx=0 tx=0).stop 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 shutdown_connections 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2e800b65f0 0x7f2e800b8b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e98109df0 0x7f2e981a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2e981094a0 0x7f2e9819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 --2- 172.21.15.39:0/294751034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e98103b60 0x7f2e9819c690 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 >> 172.21.15.39:0/294751034 conn(0x7f2e9807b850 msgr2=0x7f2e98105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:55.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 shutdown_connections 2024-09-21T01:02:55.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:55.135+0000 7f2e9f800640 1 -- 172.21.15.39:0/294751034 wait complete. 2024-09-21T01:02:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:55 smithi160 ceph-mon[34696]: pgmap v371: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:55.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:55 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/294751034' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T01:02:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:55 smithi039 ceph-mon[30415]: pgmap v371: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:55 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/294751034' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T01:02:55.988 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T01:02:55.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:02:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:55 smithi150 ceph-mon[34945]: pgmap v371: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:02:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:55 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/294751034' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T01:02:56.365 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:57.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.058+0000 7fa4d9000640 1 -- 172.21.15.39:0/117910077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d410de10 msgr2=0x7fa4d4110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:57.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.058+0000 7fa4d9000640 1 --2- 172.21.15.39:0/117910077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d410de10 0x7fa4d4110300 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009ca0 tx=0x7fa4cc02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 -- 172.21.15.39:0/117910077 shutdown_connections 2024-09-21T01:02:57.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 --2- 172.21.15.39:0/117910077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d410de10 0x7fa4d4110300 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 --2- 172.21.15.39:0/117910077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d4105300 0x7fa4d410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 --2- 172.21.15.39:0/117910077 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d4104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 -- 172.21.15.39:0/117910077 >> 172.21.15.39:0/117910077 conn(0x7fa4d4079760 msgr2=0x7fa4d4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:57.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 -- 172.21.15.39:0/117910077 shutdown_connections 2024-09-21T01:02:57.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.059+0000 7fa4d9000640 1 -- 172.21.15.39:0/117910077 wait complete. 2024-09-21T01:02:57.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.060+0000 7fa4d9000640 1 Processor -- start 2024-09-21T01:02:57.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.060+0000 7fa4d9000640 1 -- start start 2024-09-21T01:02:57.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.060+0000 7fa4d9000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:57.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.060+0000 7fa4d9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d4105300 0x7fa4d419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:57.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.060+0000 7fa4d9000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d410de10 0x7fa4d419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:57.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4113a90 con 0x7fa4d4105300 2024-09-21T01:02:57.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d9000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4113910 con 0x7fa4d410de10 2024-09-21T01:02:57.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d9000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4113c10 con 0x7fa4d41049b0 2024-09-21T01:02:57.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d4105300 0x7fa4d419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:57.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:57.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d419a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48854/0 (socket says 172.21.15.39:48854) 2024-09-21T01:02:57.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1e00640 1 -- 172.21.15.39:0/2558600115 learned_addr learned my addr 172.21.15.39:0/2558600115 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:57.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d2800640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d410de10 0x7fa4d419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:57.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1400640 1 -- 172.21.15.39:0/2558600115 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 msgr2=0x7fa4d419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:57.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1400640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1400640 1 -- 172.21.15.39:0/2558600115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d410de10 msgr2=0x7fa4d419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:57.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1400640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d410de10 0x7fa4d419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1400640 1 -- 172.21.15.39:0/2558600115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d41a03d0 con 0x7fa4d4105300 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.061+0000 7fa4d1e00640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.062+0000 7fa4d1400640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d4105300 0x7fa4d419aa00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa4d406b950 tx=0x7fa4c000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.062+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000cc40 con 0x7fa4d4105300 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.062+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4c000cde0 con 0x7fa4d4105300 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.062+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d41a06c0 con 0x7fa4d4105300 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.062+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0018780 con 0x7fa4d4105300 2024-09-21T01:02:57.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.062+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d41a6e20 con 0x7fa4d4105300 2024-09-21T01:02:57.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.064+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d406c2c0 con 0x7fa4d4105300 2024-09-21T01:02:57.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.066+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fa4c0010a90 con 0x7fa4d4105300 2024-09-21T01:02:57.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.067+0000 7fa4caa00640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4b80b6670 0x7fa4b80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:57.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.067+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4c00e7c00 con 0x7fa4d4105300 2024-09-21T01:02:57.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.068+0000 7fa4d1e00640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4b80b6670 0x7fa4b80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:57.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.068+0000 7fa4d1e00640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4b80b6670 0x7fa4b80b8b80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4011500 tx=0x7fa4c4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:57.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.071+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa4c00b2210 con 0x7fa4d4105300 2024-09-21T01:02:57.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.310+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4d41a7050 con 0x7fa4d4105300 2024-09-21T01:02:57.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.311+0000 7fa4caa00640 1 -- 172.21.15.39:0/2558600115 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11911 (secure 0 0 0) 0x7fa4c00b7150 con 0x7fa4d4105300 2024-09-21T01:02:57.312 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:57.314 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"btime":"2024-09-21T01:01:44:370343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:01:43.492789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:02:57.314 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:02:57.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4b80b6670 msgr2=0x7fa4b80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:57.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4b80b6670 0x7fa4b80b8b80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4011500 tx=0x7fa4c4005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:57.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d4105300 msgr2=0x7fa4d419aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:57.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d4105300 0x7fa4d419aa00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa4d406b950 tx=0x7fa4c000ee50 comp rx=0 tx=0).stop 2024-09-21T01:02:57.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 shutdown_connections 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4b80b6670 0x7fa4b80b8b80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d410de10 0x7fa4d419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4d4105300 0x7fa4d419aa00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 --2- 172.21.15.39:0/2558600115 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4d41049b0 0x7fa4d419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 >> 172.21.15.39:0/2558600115 conn(0x7fa4d4079760 msgr2=0x7fa4d4107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 shutdown_connections 2024-09-21T01:02:57.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:57.314+0000 7fa4d9000640 1 -- 172.21.15.39:0/2558600115 wait complete. 2024-09-21T01:02:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:57 smithi160 ceph-mon[34696]: pgmap v372: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:57 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2558600115' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:02:57.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:57 smithi150 ceph-mon[34945]: pgmap v372: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:57.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:57 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2558600115' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:02:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:57 smithi039 ceph-mon[30415]: pgmap v372: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:02:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:57 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2558600115' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:02:58.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:02:58.521 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:02:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 -- 172.21.15.39:0/655338333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 msgr2=0x7f2cf0112350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/655338333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf0112350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009ca0 tx=0x7f2ce802a8c0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 -- 172.21.15.39:0/655338333 shutdown_connections 2024-09-21T01:02:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/655338333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf0112350 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/655338333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cf0076100 0x7f2cf010f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/655338333 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2cf00757b0 0x7f2cf0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 -- 172.21.15.39:0/655338333 >> 172.21.15.39:0/655338333 conn(0x7f2cf00ff440 msgr2=0x7f2cf0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 -- 172.21.15.39:0/655338333 shutdown_connections 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.168+0000 7f2cf6e00640 1 -- 172.21.15.39:0/655338333 wait complete. 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.169+0000 7f2cf6e00640 1 Processor -- start 2024-09-21T01:02:59.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.169+0000 7f2cf6e00640 1 -- start start 2024-09-21T01:02:59.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cf00757b0 0x7f2cf019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:59.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2cf0076100 0x7f2cf019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:59.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf01a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:59.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0115ac0 con 0x7f2cf010ff10 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0115940 con 0x7f2cf00757b0 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0115c40 con 0x7f2cf0076100 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf01a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf01a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43172/0 (socket says 172.21.15.39:43172) 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 -- 172.21.15.39:0/1593455063 learned_addr learned my addr 172.21.15.39:0/1593455063 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf5a00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cf00757b0 0x7f2cf019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf5000640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2cf0076100 0x7f2cf019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 -- 172.21.15.39:0/1593455063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2cf0076100 msgr2=0x7f2cf019ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2cf0076100 0x7f2cf019ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 -- 172.21.15.39:0/1593455063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cf00757b0 msgr2=0x7f2cf019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cf00757b0 0x7f2cf019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.170+0000 7f2cf6400640 1 -- 172.21.15.39:0/1593455063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf01a2460 con 0x7f2cf010ff10 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.171+0000 7f2cf6400640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf01a1d30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8005cc0 tx=0x7f2ce8031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.171+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8003b70 con 0x7f2cf010ff10 2024-09-21T01:02:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.171+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce80318f0 con 0x7f2cf010ff10 2024-09-21T01:02:59.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.171+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf01a26f0 con 0x7f2cf010ff10 2024-09-21T01:02:59.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.171+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce802f950 con 0x7f2cf010ff10 2024-09-21T01:02:59.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.171+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf01a8eb0 con 0x7f2cf010ff10 2024-09-21T01:02:59.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.172+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf0075bc0 con 0x7f2cf010ff10 2024-09-21T01:02:59.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.175+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2ce802fb30 con 0x7f2cf010ff10 2024-09-21T01:02:59.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.176+0000 7f2cee000640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2cd80b65f0 0x7f2cd80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:02:59.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.176+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2ce8108d10 con 0x7f2cf010ff10 2024-09-21T01:02:59.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.176+0000 7f2cf5a00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2cd80b65f0 0x7f2cd80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:02:59.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.177+0000 7f2cf5a00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2cd80b65f0 0x7f2cd80b8b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009bd0 tx=0x7f2ce4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:02:59.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.180+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2ce810e020 con 0x7f2cf010ff10 2024-09-21T01:02:59.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.420+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cf01a9100 con 0x7f2cf010ff10 2024-09-21T01:02:59.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.421+0000 7f2cee000640 1 -- 172.21.15.39:0/1593455063 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11911 (secure 0 0 0) 0x7f2ce80cef40 con 0x7f2cf010ff10 2024-09-21T01:02:59.422 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:02:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"btime":"2024-09-21T01:01:44:370343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:01:43.492789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:02:59.423 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:02:59.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2cd80b65f0 msgr2=0x7f2cd80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:59.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2cd80b65f0 0x7f2cd80b8b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009bd0 tx=0x7f2ce4005f90 comp rx=0 tx=0).stop 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 msgr2=0x7f2cf01a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf01a1d30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8005cc0 tx=0x7f2ce8031700 comp rx=0 tx=0).stop 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 shutdown_connections 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cf010ff10 0x7f2cf01a1d30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2cf0076100 0x7f2cf019ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2cd80b65f0 0x7f2cd80b8b00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 --2- 172.21.15.39:0/1593455063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cf00757b0 0x7f2cf019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 >> 172.21.15.39:0/1593455063 conn(0x7f2cf00ff440 msgr2=0x7f2cf0109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 shutdown_connections 2024-09-21T01:02:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:02:59.424+0000 7f2cf6e00640 1 -- 172.21.15.39:0/1593455063 wait complete. 2024-09-21T01:02:59.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:59 smithi039 ceph-mon[30415]: pgmap v373: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:02:59.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:02:59 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1593455063' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:02:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:59 smithi160 ceph-mon[34696]: pgmap v373: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:02:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:02:59 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1593455063' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:59 smithi150 ceph-mon[34945]: pgmap v373: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:02:59 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1593455063' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:00.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:03:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:01.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 -- 172.21.15.39:0/3976900232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3948110120 msgr2=0x7f3948112540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:01.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 --2- 172.21.15.39:0/3976900232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3948110120 0x7f3948112540 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3940009ca0 tx=0x7f394002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 -- 172.21.15.39:0/3976900232 shutdown_connections 2024-09-21T01:03:01.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 --2- 172.21.15.39:0/3976900232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3948110120 0x7f3948112540 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 --2- 172.21.15.39:0/3976900232 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39480760a0 0x7f394810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 --2- 172.21.15.39:0/3976900232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f3948075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 -- 172.21.15.39:0/3976900232 >> 172.21.15.39:0/3976900232 conn(0x7f394807b850 msgr2=0x7f394807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:01.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 -- 172.21.15.39:0/3976900232 shutdown_connections 2024-09-21T01:03:01.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.285+0000 7f394fa00640 1 -- 172.21.15.39:0/3976900232 wait complete. 2024-09-21T01:03:01.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.286+0000 7f394fa00640 1 Processor -- start 2024-09-21T01:03:01.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.286+0000 7f394fa00640 1 -- start start 2024-09-21T01:03:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394fa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f394819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394fa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39480760a0 0x7f394819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394fa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3948110120 0x7f39481a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394fa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3948115ce0 con 0x7f39480760a0 2024-09-21T01:03:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394fa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3948115b60 con 0x7f3948075750 2024-09-21T01:03:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394fa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3948115e60 con 0x7f3948110120 2024-09-21T01:03:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f3947e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39480760a0 0x7f394819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f394819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f394819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46642/0 (socket says 172.21.15.39:46642) 2024-09-21T01:03:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3948110120 0x7f39481a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.287+0000 7f394d000640 1 -- 172.21.15.39:0/1822408615 learned_addr learned my addr 172.21.15.39:0/1822408615 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f394d000640 1 -- 172.21.15.39:0/1822408615 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3948110120 msgr2=0x7f39481a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f394d000640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3948110120 0x7f39481a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f394d000640 1 -- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39480760a0 msgr2=0x7f394819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f394d000640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39480760a0 0x7f394819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f394d000640 1 -- 172.21.15.39:0/1822408615 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39481a25e0 con 0x7f3948075750 2024-09-21T01:03:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f394d000640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f394819c670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f393c00d950 tx=0x7f393c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.288+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c017070 con 0x7f3948075750 2024-09-21T01:03:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.289+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f393c009c30 con 0x7f3948075750 2024-09-21T01:03:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.289+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39481a28d0 con 0x7f3948075750 2024-09-21T01:03:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.289+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c00f620 con 0x7f3948075750 2024-09-21T01:03:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.289+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39481a9030 con 0x7f3948075750 2024-09-21T01:03:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.290+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3948075b60 con 0x7f3948075750 2024-09-21T01:03:01.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.293+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f393c005030 con 0x7f3948075750 2024-09-21T01:03:01.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.295+0000 7f3945600640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39300b65f0 0x7f39300b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:01.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.295+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f393c0e7550 con 0x7f3948075750 2024-09-21T01:03:01.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.295+0000 7f3947e00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39300b65f0 0x7f39300b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:01.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.296+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f393c009de0 con 0x7f3948075750 2024-09-21T01:03:01.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.296+0000 7f3947e00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39300b65f0 0x7f39300b8b00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3948111d60 tx=0x7f393400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:01.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.543+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39481a9320 con 0x7f3948075750 2024-09-21T01:03:01.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.545+0000 7f3945600640 1 -- 172.21.15.39:0/1822408615 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25331 (secure 0 0 0) 0x7f393c0b1b60 con 0x7f3948075750 2024-09-21T01:03:01.546 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:01.547 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":90,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:02:10.480352+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:03:01.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39300b65f0 msgr2=0x7f39300b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:01.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39300b65f0 0x7f39300b8b00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3948111d60 tx=0x7f393400b410 comp rx=0 tx=0).stop 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 msgr2=0x7f394819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f394819c670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f393c00d950 tx=0x7f393c00de10 comp rx=0 tx=0).stop 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 shutdown_connections 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3948110120 0x7f39481a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39480760a0 0x7f394819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39300b65f0 0x7f39300b8b00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 --2- 172.21.15.39:0/1822408615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3948075750 0x7f394819c670 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 >> 172.21.15.39:0/1822408615 conn(0x7f394807b850 msgr2=0x7f3948100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 shutdown_connections 2024-09-21T01:03:01.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:01.547+0000 7f394fa00640 1 -- 172.21.15.39:0/1822408615 wait complete. 2024-09-21T01:03:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:01 smithi160 ceph-mon[34696]: pgmap v374: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:01 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1822408615' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:03:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:01 smithi039 ceph-mon[30415]: pgmap v374: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:01 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1822408615' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:03:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:01 smithi150 ceph-mon[34945]: pgmap v374: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:01 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1822408615' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:03:02.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:03:02.756 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 -- 172.21.15.39:0/4009438181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc1094a0 msgr2=0x7f1dfc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 --2- 172.21.15.39:0/4009438181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc1094a0 0x7f1dfc1098b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009900 tx=0x7f1de402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 -- 172.21.15.39:0/4009438181 shutdown_connections 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 --2- 172.21.15.39:0/4009438181 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1dfc109df0 0x7f1dfc110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 --2- 172.21.15.39:0/4009438181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc1094a0 0x7f1dfc1098b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 --2- 172.21.15.39:0/4009438181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dfc103b60 0x7f1dfc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 -- 172.21.15.39:0/4009438181 >> 172.21.15.39:0/4009438181 conn(0x7f1dfc07b850 msgr2=0x7f1dfc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 -- 172.21.15.39:0/4009438181 shutdown_connections 2024-09-21T01:03:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.428+0000 7f1e01c00640 1 -- 172.21.15.39:0/4009438181 wait complete. 2024-09-21T01:03:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.429+0000 7f1e01c00640 1 Processor -- start 2024-09-21T01:03:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.429+0000 7f1e01c00640 1 -- start start 2024-09-21T01:03:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1e01c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 0x7f1dfc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1e01c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dfc1094a0 0x7f1dfc19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1e01c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1dfc109df0 0x7f1dfc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1e01c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc113ab0 con 0x7f1dfc103b60 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1e01c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc113930 con 0x7f1dfc1094a0 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1e01c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc113c30 con 0x7f1dfc109df0 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 0x7f1dfc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 0x7f1dfc19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57744/0 (socket says 172.21.15.39:57744) 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 -- 172.21.15.39:0/1187884148 learned_addr learned my addr 172.21.15.39:0/1187884148 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfb400640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1dfc109df0 0x7f1dfc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:03.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1df3e00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dfc1094a0 0x7f1dfc19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 -- 172.21.15.39:0/1187884148 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1dfc109df0 msgr2=0x7f1dfc19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1dfc109df0 0x7f1dfc19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 -- 172.21.15.39:0/1187884148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dfc1094a0 msgr2=0x7f1dfc19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.430+0000 7f1dfaa00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dfc1094a0 0x7f1dfc19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1dfaa00640 1 -- 172.21.15.39:0/1187884148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc1a03f0 con 0x7f1dfc103b60 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1dfaa00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 0x7f1dfc19a480 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e9f0 tx=0x7f1dec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00cd80 con 0x7f1dfc103b60 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dec0108e0 con 0x7f1dfc103b60 2024-09-21T01:03:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc1a06e0 con 0x7f1dfc103b60 2024-09-21T01:03:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec018980 con 0x7f1dfc103b60 2024-09-21T01:03:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.431+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc1a6e40 con 0x7f1dfc103b60 2024-09-21T01:03:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.433+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dfc103f70 con 0x7f1dfc103b60 2024-09-21T01:03:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.436+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f1dec010430 con 0x7f1dfc103b60 2024-09-21T01:03:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.437+0000 7f1df3400640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1de00b65f0 0x7f1de00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.437+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1dec0e7ce0 con 0x7f1dfc103b60 2024-09-21T01:03:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.438+0000 7f1df3e00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1de00b65f0 0x7f1de00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.438+0000 7f1df3e00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1de00b65f0 0x7f1de00b8b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009230 tx=0x7f1de4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.441+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1dec0b22f0 con 0x7f1dfc103b60 2024-09-21T01:03:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.683+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dfc19b5b0 con 0x7f1dfc103b60 2024-09-21T01:03:03.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.684+0000 7f1df3400640 1 -- 172.21.15.39:0/1187884148 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11911 (secure 0 0 0) 0x7f1dec0b7230 con 0x7f1dfc103b60 2024-09-21T01:03:03.685 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:03.685 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"btime":"2024-09-21T01:01:44:370343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:01:43.492789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:03:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1de00b65f0 msgr2=0x7f1de00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1de00b65f0 0x7f1de00b8b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009230 tx=0x7f1de4031400 comp rx=0 tx=0).stop 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 msgr2=0x7f1dfc19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 0x7f1dfc19a480 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e9f0 tx=0x7f1dec00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 shutdown_connections 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1dfc109df0 0x7f1dfc19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1de00b65f0 0x7f1de00b8b00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dfc1094a0 0x7f1dfc19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 --2- 172.21.15.39:0/1187884148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dfc103b60 0x7f1dfc19a480 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:03.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.686+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 >> 172.21.15.39:0/1187884148 conn(0x7f1dfc07b850 msgr2=0x7f1dfc107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.687+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 shutdown_connections 2024-09-21T01:03:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:03.687+0000 7f1e01c00640 1 -- 172.21.15.39:0/1187884148 wait complete. 2024-09-21T01:03:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:03 smithi039 ceph-mon[30415]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:03 smithi150 ceph-mon[34945]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:04.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:03 smithi160 ceph-mon[34696]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:04.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T01:00:43.758917+0000', 'modified': '2024-09-21T01:01:43.492789+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24764, 'mds_1': 24760, 'mds_2': 24742}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24715': {'gid': 24715, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6835/2680876155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6834', 'nonce': 2680876155}, {'type': 'v1', 'addr': '172.21.15.160:6835', 'nonce': 2680876155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24725': {'gid': 24725, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/555411539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 555411539}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 555411539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24727': {'gid': 24727, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6837/1965195431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6836', 'nonce': 1965195431}, {'type': 'v1', 'addr': '172.21.15.160:6837', 'nonce': 1965195431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24742': {'gid': 24742, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.160:6839/2403371273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6838', 'nonce': 2403371273}, {'type': 'v1', 'addr': '172.21.15.160:6839', 'nonce': 2403371273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24760': {'gid': 24760, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.39:6841/3868264331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6840', 'nonce': 3868264331}, {'type': 'v1', 'addr': '172.21.15.39:6841', 'nonce': 3868264331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24764': {'gid': 24764, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.150:6839/103174700', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 103174700}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 103174700}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24764, 'qdb_cluster': [24764, 24760, 24742]} 2024-09-21T01:03:04.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-21T01:03:04.493 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T01:03:04.875 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:04 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1187884148' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:05.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:04 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1187884148' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:05.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:04 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1187884148' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 -- 172.21.15.39:0/3618329396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 msgr2=0x7f97001098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 --2- 172.21.15.39:0/3618329396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f97001098b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009900 tx=0x7f96e802a410 comp rx=0 tx=0).stop 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 -- 172.21.15.39:0/3618329396 shutdown_connections 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 --2- 172.21.15.39:0/3618329396 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9700109df0 0x7f9700110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 --2- 172.21.15.39:0/3618329396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f97001098b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 --2- 172.21.15.39:0/3618329396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9700103b60 0x7f9700103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 -- 172.21.15.39:0/3618329396 >> 172.21.15.39:0/3618329396 conn(0x7f970007b850 msgr2=0x7f970007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 -- 172.21.15.39:0/3618329396 shutdown_connections 2024-09-21T01:03:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.509+0000 7f9706a00640 1 -- 172.21.15.39:0/3618329396 wait complete. 2024-09-21T01:03:05.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.510+0000 7f9706a00640 1 Processor -- start 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 -- start start 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9700103b60 0x7f970019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f970019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9700109df0 0x7f97001a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9700113b00 con 0x7f97001094a0 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9700113980 con 0x7f9700103b60 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.511+0000 7f9706a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9700113c80 con 0x7f9700109df0 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f970019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96ffe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9700103b60 0x7f970019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f9704a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9700109df0 0x7f97001a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f970019cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57768/0 (socket says 172.21.15.39:57768) 2024-09-21T01:03:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 -- 172.21.15.39:0/3770683102 learned_addr learned my addr 172.21.15.39:0/3770683102 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:05.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 -- 172.21.15.39:0/3770683102 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9700109df0 msgr2=0x7f97001a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9700109df0 0x7f97001a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 -- 172.21.15.39:0/3770683102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9700103b60 msgr2=0x7f970019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9700103b60 0x7f970019c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 -- 172.21.15.39:0/3770683102 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97001a2620 con 0x7f97001094a0 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.512+0000 7f96f7e00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f970019cc50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96e8005ab0 tx=0x7f96e8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.513+0000 7f96fd600640 1 -- 172.21.15.39:0/3770683102 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e803f070 con 0x7f97001094a0 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.513+0000 7f96fd600640 1 -- 172.21.15.39:0/3770683102 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96e8031b50 con 0x7f97001094a0 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.513+0000 7f96fd600640 1 -- 172.21.15.39:0/3770683102 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8035030 con 0x7f97001094a0 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.513+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97001a28b0 con 0x7f97001094a0 2024-09-21T01:03:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.513+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97001a9070 con 0x7f97001094a0 2024-09-21T01:03:05.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.516+0000 7f970d7d4740 1 Processor -- start 2024-09-21T01:03:05.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.516+0000 7f970d7d4740 1 -- start start 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08817490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b088179d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08817f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f970d7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555b0878ed50 con 0x555b08450280 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f970d7d4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555b0877e180 con 0x555b084480a0 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f970d7d4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555b08781f90 con 0x555b08470f60 2024-09-21T01:03:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f96fd600640 1 -- 172.21.15.39:0/3770683102 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f96e8042070 con 0x7f97001094a0 2024-09-21T01:03:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f9704a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08817490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f96ffe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b088179d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f9704a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08817490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42084/0 (socket says 172.21.15.39:42084) 2024-09-21T01:03:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f9704a00640 1 -- 172.21.15.39:0/2359839221 learned_addr learned my addr 172.21.15.39:0/2359839221 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.517+0000 7f96f7e00640 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08817f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f9704a00640 1 -- 172.21.15.39:0/2359839221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 msgr2=0x555b088179d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f9704a00640 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b088179d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f9704a00640 1 -- 172.21.15.39:0/2359839221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 msgr2=0x555b08817f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f9704a00640 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08817f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f9704a00640 1 -- 172.21.15.39:0/2359839221 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b08748a70 con 0x555b08470f60 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f9704a00640 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08817490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009fd0 tx=0x7f96f80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3097c7fcb736b9a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f96f5200640 1 -- 172.21.15.39:0/2359839221 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f8004d20 con 0x555b08470f60 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f96f5200640 1 -- 172.21.15.39:0/2359839221 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96f8004050 con 0x555b08470f60 2024-09-21T01:03:05.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.518+0000 7f96f5200640 1 -- 172.21.15.39:0/2359839221 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f8005280 con 0x555b08470f60 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f96fd600640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f96e40b65f0 0x7f96e40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f96fd600640 1 -- 172.21.15.39:0/3770683102 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f96e8107be0 con 0x7f97001094a0 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f96ffe00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f96e40b65f0 0x7f96e40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 -- 172.21.15.39:0/2359839221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 msgr2=0x555b08817490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08817490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009fd0 tx=0x7f96f80118d0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 -- 172.21.15.39:0/2359839221 shutdown_connections 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08817f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b088179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2359839221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08817490 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.519+0000 7f970d7d4740 1 -- 172.21.15.39:0/2359839221 >> 172.21.15.39:0/2359839221 conn(0x555b08465350 msgr2=0x555b08815380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.520+0000 7f970d7d4740 1 -- 172.21.15.39:0/2359839221 shutdown_connections 2024-09-21T01:03:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.520+0000 7f970d7d4740 1 -- 172.21.15.39:0/2359839221 wait complete. 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.520+0000 7f96ffe00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f96e40b65f0 0x7f96e40b8b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x555b08816110 tx=0x7f96f0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.520+0000 7f970d7d4740 1 -- start start 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 Processor -- start 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b08822ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b0882efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555b085df5f0 con 0x555b08450280 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555b08719460 con 0x555b084480a0 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555b08825c70 con 0x555b08470f60 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f970d7d4740 10 client.0 _ll_register_callbacks cb 0x555b08729210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.521+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57784/0 (socket says 172.21.15.39:57784) 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 -- 172.21.15.39:0/764779064 learned_addr learned my addr 172.21.15.39:0/764779064 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:05.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f9704a00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b0882efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96ffe00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b08822ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 -- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 msgr2=0x555b0882efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b0882efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 -- 172.21.15.39:0/764779064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 msgr2=0x555b08822ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b08822ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 -- 172.21.15.39:0/764779064 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b084dc540 con 0x555b08450280 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f9704a00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b0882efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96f7e00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823530 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f96e80cd4c0 tx=0x7f96e8031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca6b8f4b8d84bde server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96ed600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.522+0000 7f96ed600640 1 -- 172.21.15.39:0/764779064 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8049070 con 0x555b08450280 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f96ed600640 1 -- 172.21.15.39:0/764779064 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f96e80d6ca0 con 0x555b08450280 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b0871ba70 con 0x555b08450280 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f96ed600640 1 -- 172.21.15.39:0/764779064 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96e80cd110 con 0x555b08450280 2024-09-21T01:03:05.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 10 client.15198 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555b087366f0 con 0x555b08450280 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f96ed600640 1 -- 172.21.15.39:0/764779064 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555b0871ba70 con 0x555b08450280 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f96ed600640 1 -- 172.21.15.39:0/764779064 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f96e804c060 con 0x555b08450280 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 10 client.15198 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 10 client.15198 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 20 client.15198 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.523+0000 7f970d7d4740 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b0842f8e0 0x555b088355e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.524+0000 7f970d7d4740 4 client.15198 mds_command: new command op to 24742 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.524+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 --> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b08790ca0 con 0x555b0842f8e0 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.524+0000 7f96ffe00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b0842f8e0 0x555b088355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.524+0000 7f96ffe00640 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b0842f8e0 0x555b088355e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7633c1d6a72c242 server_cookie=1bb11750201f4cd4 in_seq=0 out_seq=0 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.524+0000 7f96ed600640 10 client.15198 ms_handle_connect on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.525+0000 7f96ed600640 1 -- 172.21.15.39:0/764779064 <== mds.2 v2:172.21.15.160:6838/2403371273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555b08790ca0 con 0x555b0842f8e0 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.525+0000 7f96ed600640 10 client.15198 handle_command_reply: tid=0 2024-09-21T01:03:05.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.525+0000 7f970d7d4740 1 client.15198 shutdown 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 msgr2=0x555b08823530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823530 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f96e80cd4c0 tx=0x7f96e8031f50 comp rx=0 tx=0).stop 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 shutdown_connections 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b0842f8e0 0x555b088355e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b0882efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823530 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 --2- 172.21.15.39:0/764779064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b08822ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 >> 172.21.15.39:0/764779064 conn(0x555b08465350 msgr2=0x555b0881a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f96ed600640 0 client.15198 ms_handle_reset on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 shutdown_connections 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 1 -- 172.21.15.39:0/764779064 wait complete. 2024-09-21T01:03:05.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.526+0000 7f970d7d4740 20 client.15198 trim_cache size 0 max 16384 2024-09-21T01:03:05.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.545+0000 7f970d7d4740 1 Processor -- start 2024-09-21T01:03:05.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.545+0000 7f970d7d4740 1 -- start start 2024-09-21T01:03:05.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b084480a0 0x555b08822fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b08470f60 0x555b08821670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f970d7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555b08833920 con 0x555b08450280 2024-09-21T01:03:05.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f970d7d4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555b08833ff0 con 0x555b08470f60 2024-09-21T01:03:05.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f970d7d4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555b088346c0 con 0x555b084480a0 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96ffe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b084480a0 0x555b08822fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57796/0 (socket says 172.21.15.39:57796) 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96f7e00640 1 -- 172.21.15.39:0/3233429018 learned_addr learned my addr 172.21.15.39:0/3233429018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f9704a00640 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b08470f60 0x555b08821670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96ffe00640 1 -- 172.21.15.39:0/3233429018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b08470f60 msgr2=0x555b08821670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96ffe00640 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b08470f60 0x555b08821670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96ffe00640 1 -- 172.21.15.39:0/3233429018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 msgr2=0x555b08823510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96ffe00640 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.546+0000 7f96ffe00640 1 -- 172.21.15.39:0/3233429018 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b0881da90 con 0x555b084480a0 2024-09-21T01:03:05.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.547+0000 7f96f7e00640 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:05.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.547+0000 7f96ffe00640 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b084480a0 0x555b08822fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x555b08823cf0 tx=0x7f96f001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f90a9dd66bda3518 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.547+0000 7f96efe00640 1 -- 172.21.15.39:0/3233429018 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0005020 con 0x555b084480a0 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.547+0000 7f96efe00640 1 -- 172.21.15.39:0/3233429018 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96f00047a0 con 0x555b084480a0 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.547+0000 7f96efe00640 1 -- 172.21.15.39:0/3233429018 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f002b6d0 con 0x555b084480a0 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 -- 172.21.15.39:0/3233429018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b084480a0 msgr2=0x555b08822fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b084480a0 0x555b08822fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x555b08823cf0 tx=0x7f96f001b000 comp rx=0 tx=0).stop 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 -- 172.21.15.39:0/3233429018 shutdown_connections 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b08470f60 0x555b08821670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08823510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 --2- 172.21.15.39:0/3233429018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b084480a0 0x555b08822fd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 -- 172.21.15.39:0/3233429018 >> 172.21.15.39:0/3233429018 conn(0x555b08465350 msgr2=0x555b0881dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:05.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 -- 172.21.15.39:0/3233429018 shutdown_connections 2024-09-21T01:03:05.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.548+0000 7f970d7d4740 1 -- 172.21.15.39:0/3233429018 wait complete. 2024-09-21T01:03:05.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 -- start start 2024-09-21T01:03:05.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 Processor -- start 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b0883e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08837530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08837a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555b08835460 con 0x555b08450280 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555b08834d90 con 0x555b084480a0 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.549+0000 7f970d7d4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555b08835b30 con 0x555b08470f60 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f970d7d4740 10 client.0 _ll_register_callbacks cb 0x7f96e8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08837530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96ffe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b0883e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96ffe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b0883e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:47928/0 (socket says 172.21.15.39:47928) 2024-09-21T01:03:05.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96ffe00640 1 -- 172.21.15.39:0/2062724214 learned_addr learned my addr 172.21.15.39:0/2062724214 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f9704a00640 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08837a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96f7e00640 1 -- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 msgr2=0x555b08837a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96f7e00640 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08837a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96f7e00640 1 -- 172.21.15.39:0/2062724214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 msgr2=0x555b0883e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96f7e00640 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b0883e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.550+0000 7f96f7e00640 1 -- 172.21.15.39:0/2062724214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b088381e0 con 0x555b08450280 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f96f7e00640 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08837530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x555b08819b70 tx=0x7f96e802a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1db3287f43aa4e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f96ed600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f96ed600640 1 -- 172.21.15.39:0/2062724214 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8049070 con 0x555b08450280 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f96ed600640 1 -- 172.21.15.39:0/2062724214 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f96e80316a0 con 0x555b08450280 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f00047a0 con 0x555b08450280 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f96ed600640 1 -- 172.21.15.39:0/2062724214 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96e8107e60 con 0x555b08450280 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f970d7d4740 10 client.15204 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:05.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555b088a0ab0 con 0x555b08450280 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.551+0000 7f96ed600640 1 -- 172.21.15.39:0/2062724214 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96f00047a0 con 0x555b08450280 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f96ed600640 1 -- 172.21.15.39:0/2062724214 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f96e80cc490 con 0x555b08450280 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f970d7d4740 10 client.15204 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f970d7d4740 10 client.15204 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f970d7d4740 20 client.15204 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b08404c90 0x555b088a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f970d7d4740 4 client.15204 mds_command: new command op to 24742 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:03:05.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 --> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] -- command(tid 0: {"prefix": "status"}) -- 0x555b0878c750 con 0x555b08404c90 2024-09-21T01:03:05.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.552+0000 7f96ffe00640 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b08404c90 0x555b088a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:05.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f96ffe00640 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b08404c90 0x555b088a4080 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ef21ec497bd7e68 server_cookie=149abdb9a3d11ae8 in_seq=0 out_seq=0 2024-09-21T01:03:05.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f96ed600640 10 client.15204 ms_handle_connect on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:05.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f96ed600640 1 -- 172.21.15.39:0/2062724214 <== mds.2 v2:172.21.15.160:6838/2403371273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555b0878c750 con 0x555b08404c90 2024-09-21T01:03:05.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f96ed600640 10 client.15204 handle_command_reply: tid=0 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f970d7d4740 1 client.15204 shutdown 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 msgr2=0x555b08837530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08837530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x555b08819b70 tx=0x7f96e802a9b0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 shutdown_connections 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x555b08404c90 0x555b088a4080 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.553+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555b08470f60 0x555b08837a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f96ed600640 0 client.15204 ms_handle_reset on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555b08450280 0x555b08837530 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f970d7d4740 1 --2- 172.21.15.39:0/2062724214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555b084480a0 0x555b0883e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 >> 172.21.15.39:0/2062724214 conn(0x555b08465350 msgr2=0x555b08822fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 shutdown_connections 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f970d7d4740 1 -- 172.21.15.39:0/2062724214 wait complete. 2024-09-21T01:03:05.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.554+0000 7f970d7d4740 20 client.15204 trim_cache size 0 max 16384 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 2, 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24742, 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 87.213840110999996, 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:03:05.558 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 166.20911012799999 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f96e40b65f0 msgr2=0x7f96e40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f96e40b65f0 0x7f96e40b8b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x555b08816110 tx=0x7f96f0005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 msgr2=0x7f970019cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f970019cc50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96e8005ab0 tx=0x7f96e8003b90 comp rx=0 tx=0).stop 2024-09-21T01:03:05.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 shutdown_connections 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9700109df0 0x7f97001a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97001094a0 0x7f970019cc50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f96e40b65f0 0x7f96e40b8b00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 --2- 172.21.15.39:0/3770683102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9700103b60 0x7f970019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 >> 172.21.15.39:0/3770683102 conn(0x7f970007b850 msgr2=0x7f9700107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 shutdown_connections 2024-09-21T01:03:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:05.555+0000 7f9706a00640 1 -- 172.21.15.39:0/3770683102 wait complete. 2024-09-21T01:03:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:05 smithi039 ceph-mon[30415]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:05 smithi150 ceph-mon[34945]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:05 smithi160 ceph-mon[34696]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:06.474 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.a status 2024-09-21T01:03:06.858 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:07.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.507+0000 7fc4af200640 1 -- 172.21.15.39:0/823732284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b01092f0 msgr2=0x7fc4b0109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.507+0000 7fc4af200640 1 --2- 172.21.15.39:0/823732284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b01092f0 0x7fc4b0109700 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009900 tx=0x7fc49c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 -- 172.21.15.39:0/823732284 shutdown_connections 2024-09-21T01:03:07.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 --2- 172.21.15.39:0/823732284 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b0109c40 0x7fc4b0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 --2- 172.21.15.39:0/823732284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b01092f0 0x7fc4b0109700 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 --2- 172.21.15.39:0/823732284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01039b0 0x7fc4b0103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 -- 172.21.15.39:0/823732284 >> 172.21.15.39:0/823732284 conn(0x7fc4b00ff400 msgr2=0x7fc4b0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:07.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 -- 172.21.15.39:0/823732284 shutdown_connections 2024-09-21T01:03:07.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.508+0000 7fc4af200640 1 -- 172.21.15.39:0/823732284 wait complete. 2024-09-21T01:03:07.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 Processor -- start 2024-09-21T01:03:07.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 -- start start 2024-09-21T01:03:07.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 0x7fc4b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01092f0 0x7fc4b0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b0109c40 0x7fc4b0076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0113900 con 0x7fc4b0109c40 2024-09-21T01:03:07.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0113780 con 0x7fc4b01092f0 2024-09-21T01:03:07.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.509+0000 7fc4af200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0113a80 con 0x7fc4b01039b0 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b0109c40 0x7fc4b0076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 0x7fc4b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01092f0 0x7fc4b0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01092f0 0x7fc4b0075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:47952/0 (socket says 172.21.15.39:47952) 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 0x7fc4b0075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42136/0 (socket says 172.21.15.39:42136) 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b0109c40 0x7fc4b0076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57814/0 (socket says 172.21.15.39:57814) 2024-09-21T01:03:07.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ad400640 1 -- 172.21.15.39:0/688566326 learned_addr learned my addr 172.21.15.39:0/688566326 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 -- 172.21.15.39:0/688566326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01092f0 msgr2=0x7fc4b0075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01092f0 0x7fc4b0075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 -- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b0109c40 msgr2=0x7fc4b0076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b0109c40 0x7fc4b0076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.510+0000 7fc4ade00640 1 -- 172.21.15.39:0/688566326 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b01a85b0 con 0x7fc4b01039b0 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.511+0000 7fc4ade00640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 0x7fc4b0075710 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000e9f0 tx=0x7fc4a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.511+0000 7fc4a6a00640 1 -- 172.21.15.39:0/688566326 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a000ccc0 con 0x7fc4b01039b0 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.511+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b01a88a0 con 0x7fc4b01039b0 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.511+0000 7fc4a6a00640 1 -- 172.21.15.39:0/688566326 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a00108e0 con 0x7fc4b01039b0 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.511+0000 7fc4a6a00640 1 -- 172.21.15.39:0/688566326 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0018880 con 0x7fc4b01039b0 2024-09-21T01:03:07.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.511+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b01a8e30 con 0x7fc4b01039b0 2024-09-21T01:03:07.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.514+0000 7fc4bbff8740 1 Processor -- start 2024-09-21T01:03:07.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.514+0000 7fc4bbff8740 1 -- start start 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 0x55d02cbf1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c82a280 0x55d02cbf20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d02cb68d50 con 0x55d02c84af60 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d02cb58180 con 0x55d02c8220a0 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d02cb5bf90 con 0x55d02c82a280 2024-09-21T01:03:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 0x55d02cbf1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 0x55d02cbf1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57822/0 (socket says 172.21.15.39:57822) 2024-09-21T01:03:07.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 -- 172.21.15.39:0/2253915408 learned_addr learned my addr 172.21.15.39:0/2253915408 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:07.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ad400640 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c82a280 0x55d02cbf20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4a6a00640 1 -- 172.21.15.39:0/688566326 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fc4a0010430 con 0x7fc4b01039b0 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 -- 172.21.15.39:0/2253915408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c82a280 msgr2=0x55d02cbf20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c82a280 0x55d02cbf20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 -- 172.21.15.39:0/2253915408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 msgr2=0x55d02cbf1bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf1bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.515+0000 7fc4ae800640 1 -- 172.21.15.39:0/2253915408 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02cb22a70 con 0x55d02c84af60 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.516+0000 7fc4ae800640 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 0x55d02cbf1670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fd0 tx=0x7fc4a80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b208736dc9deaa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.516+0000 7fc49b400640 1 -- 172.21.15.39:0/2253915408 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004d20 con 0x55d02c84af60 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.516+0000 7fc49b400640 1 -- 172.21.15.39:0/2253915408 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a8004050 con 0x55d02c84af60 2024-09-21T01:03:07.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.516+0000 7fc49b400640 1 -- 172.21.15.39:0/2253915408 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8005280 con 0x55d02c84af60 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 -- 172.21.15.39:0/2253915408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 msgr2=0x55d02cbf1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 0x55d02cbf1670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009fd0 tx=0x7fc4a80118d0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 -- 172.21.15.39:0/2253915408 shutdown_connections 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c82a280 0x55d02cbf20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/2253915408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c84af60 0x55d02cbf1670 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 -- 172.21.15.39:0/2253915408 >> 172.21.15.39:0/2253915408 conn(0x55d02c83f350 msgr2=0x55d02cbef560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4bbff8740 1 -- 172.21.15.39:0/2253915408 shutdown_connections 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4a6a00640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc4940b6620 0x7fc4940b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.517+0000 7fc4a6a00640 1 -- 172.21.15.39:0/688566326 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc4a00e7c60 con 0x7fc4b01039b0 2024-09-21T01:03:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.518+0000 7fc4bbff8740 1 -- 172.21.15.39:0/2253915408 wait complete. 2024-09-21T01:03:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.518+0000 7fc4ad400640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc4940b6620 0x7fc4940b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.518+0000 7fc4bbff8740 1 -- start start 2024-09-21T01:03:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.518+0000 7fc4bbff8740 1 Processor -- start 2024-09-21T01:03:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.518+0000 7fc4ad400640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc4940b6620 0x7fc4940b8b30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0106e60 tx=0x7fc49c005890 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 0x55d02cbfd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbfd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c84af60 0x55d02cc09140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d02c9b95f0 con 0x55d02c82a280 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d02caf3460 con 0x55d02c84af60 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d02cbffe50 con 0x55d02c8220a0 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4bbff8740 10 client.0 _ll_register_callbacks cb 0x55d02cb03210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 0x55d02cbfd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbfd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 0x55d02cbfd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42152/0 (socket says 172.21.15.39:42152) 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c84af60 0x55d02cc09140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.519+0000 7fc4ade00640 1 -- 172.21.15.39:0/3186254276 learned_addr learned my addr 172.21.15.39:0/3186254276 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4ade00640 1 -- 172.21.15.39:0/3186254276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c84af60 msgr2=0x55d02cc09140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4ade00640 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c84af60 0x55d02cc09140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4ade00640 1 -- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 msgr2=0x55d02cbfd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4ade00640 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbfd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4ade00640 1 -- 172.21.15.39:0/3186254276 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02c8b6540 con 0x55d02c8220a0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4ade00640 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 0x55d02cbfd250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00ad5a0 tx=0x7fc4a00ad0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38810c64dea60174 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc48fe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc48fe00640 1 -- 172.21.15.39:0/3186254276 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00d2060 con 0x55d02c8220a0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc48fe00640 1 -- 172.21.15.39:0/3186254276 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc4a00aed20 con 0x55d02c8220a0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.520+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d02caf5a70 con 0x55d02c8220a0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.521+0000 7fc48fe00640 1 -- 172.21.15.39:0/3186254276 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc4a00b22d0 con 0x55d02c8220a0 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.521+0000 7fc4bbff8740 10 client.25225 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:07.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.521+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d02cb106f0 con 0x55d02c8220a0 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.521+0000 7fc48fe00640 1 -- 172.21.15.39:0/3186254276 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d02caf5a70 con 0x55d02c8220a0 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.521+0000 7fc48fe00640 1 -- 172.21.15.39:0/3186254276 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc4a00ac670 con 0x55d02c8220a0 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4bbff8740 10 client.25225 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4bbff8740 10 client.25225 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4bbff8740 20 client.25225 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c8098e0 0x55d02cc0f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4bbff8740 4 client.25225 mds_command: new command op to 24760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 --> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d02cb6aca0 con 0x55d02c8098e0 2024-09-21T01:03:07.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.522+0000 7fc4ad400640 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c8098e0 0x55d02cc0f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.523+0000 7fc4ad400640 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c8098e0 0x55d02cc0f5f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4620bead0737926 server_cookie=4edc1f40efa0a347 in_seq=0 out_seq=0 2024-09-21T01:03:07.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.523+0000 7fc48fe00640 10 client.25225 ms_handle_connect on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:07.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.523+0000 7fc48fe00640 1 -- 172.21.15.39:0/3186254276 <== mds.1 v2:172.21.15.39:6840/3868264331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d02cb6aca0 con 0x55d02c8098e0 2024-09-21T01:03:07.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.523+0000 7fc48fe00640 10 client.25225 handle_command_reply: tid=0 2024-09-21T01:03:07.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 client.25225 shutdown 2024-09-21T01:03:07.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 msgr2=0x55d02cbfd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 0x55d02cbfd250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00ad5a0 tx=0x7fc4a00ad0d0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 shutdown_connections 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c8098e0 0x55d02cc0f5f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c84af60 0x55d02cc09140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbfd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3186254276 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c8220a0 0x55d02cbfd250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc48fe00640 0 client.25225 ms_handle_reset on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 >> 172.21.15.39:0/3186254276 conn(0x55d02c83f350 msgr2=0x55d02cbf4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 shutdown_connections 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3186254276 wait complete. 2024-09-21T01:03:07.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.524+0000 7fc4bbff8740 20 client.25225 trim_cache size 0 max 16384 2024-09-21T01:03:07.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.543+0000 7fc4bbff8740 1 Processor -- start 2024-09-21T01:03:07.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 -- start start 2024-09-21T01:03:07.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbf6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc090b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d02cc0d930 con 0x55d02c82a280 2024-09-21T01:03:07.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d02cc0e000 con 0x55d02c8220a0 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d02cc0e6d0 con 0x55d02c84af60 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbf6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:47972/0 (socket says 172.21.15.39:47972) 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4ade00640 1 -- 172.21.15.39:0/3699249267 learned_addr learned my addr 172.21.15.39:0/3699249267 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbf6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57844/0 (socket says 172.21.15.39:57844) 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.544+0000 7fc4ae800640 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ade00640 1 -- 172.21.15.39:0/3699249267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 msgr2=0x55d02cc090b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ade00640 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc090b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ade00640 1 -- 172.21.15.39:0/3699249267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 msgr2=0x55d02cbf6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ade00640 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbf6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ade00640 1 -- 172.21.15.39:0/3699249267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02cbfe310 con 0x55d02c8220a0 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ae800640 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc090b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc4ade00640 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf68f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00aca10 tx=0x7fc4a000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=823cd06ac43e161 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc49aa00640 1 -- 172.21.15.39:0/3699249267 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00d2070 con 0x55d02c8220a0 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc49aa00640 1 -- 172.21.15.39:0/3699249267 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a00ac570 con 0x55d02c8220a0 2024-09-21T01:03:07.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.545+0000 7fc49aa00640 1 -- 172.21.15.39:0/3699249267 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00b6a00 con 0x55d02c8220a0 2024-09-21T01:03:07.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3699249267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 msgr2=0x55d02cbf68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf68f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00aca10 tx=0x7fc4a000cef0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3699249267 shutdown_connections 2024-09-21T01:03:07.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc090b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cbf6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/3699249267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cbf68f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3699249267 >> 172.21.15.39:0/3699249267 conn(0x55d02c83f350 msgr2=0x55d02cbea2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3699249267 shutdown_connections 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.546+0000 7fc4bbff8740 1 -- 172.21.15.39:0/3699249267 wait complete. 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 -- start start 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 Processor -- start 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cc17850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cc17d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc182d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d02cc0f470 con 0x55d02c82a280 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d02cc0eda0 con 0x55d02c8220a0 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d02cc0fb40 con 0x55d02c84af60 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.547+0000 7fc4bbff8740 10 client.0 _ll_register_callbacks cb 0x7fc4a00ad2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cc17d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cc17d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57850/0 (socket says 172.21.15.39:57850) 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 -- 172.21.15.39:0/183341673 learned_addr learned my addr 172.21.15.39:0/183341673 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ade00640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cc17850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ae800640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc182d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 -- 172.21.15.39:0/183341673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 msgr2=0x55d02cc182d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc182d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 -- 172.21.15.39:0/183341673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 msgr2=0x55d02cc17850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cc17850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.548+0000 7fc4ad400640 1 -- 172.21.15.39:0/183341673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02cc10820 con 0x55d02c82a280 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc4ad400640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cc17d90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009230 tx=0x7fc49c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3f954b27c8e622 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc48fe00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc48fe00640 1 -- 172.21.15.39:0/183341673 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c05f060 con 0x55d02c82a280 2024-09-21T01:03:07.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc48fe00640 1 -- 172.21.15.39:0/183341673 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc49c0035f0 con 0x55d02c82a280 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a00ac570 con 0x55d02c82a280 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc48fe00640 1 -- 172.21.15.39:0/183341673 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc49c066050 con 0x55d02c82a280 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc4bbff8740 10 client.15222 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.549+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d02cc7adc0 con 0x55d02c82a280 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc48fe00640 1 -- 172.21.15.39:0/183341673 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a00ac570 con 0x55d02c82a280 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc48fe00640 1 -- 172.21.15.39:0/183341673 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc49c03d830 con 0x55d02c82a280 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4bbff8740 10 client.15222 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4bbff8740 10 client.15222 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4bbff8740 20 client.15222 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c7dec90 0x55d02cc7e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4bbff8740 4 client.15222 mds_command: new command op to 24760 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:03:07.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 --> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] -- command(tid 0: {"prefix": "status"}) -- 0x55d02cb66750 con 0x55d02c7dec90 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.550+0000 7fc4ade00640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c7dec90 0x55d02cc7e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc48fe00640 10 client.15222 ms_handle_connect on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc4ade00640 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c7dec90 0x55d02cc7e670 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c8267e598de3063 server_cookie=d7f756edfefe58d0 in_seq=0 out_seq=0 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc48fe00640 1 -- 172.21.15.39:0/183341673 <== mds.1 v2:172.21.15.39:6840/3868264331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d02cb66750 con 0x55d02c7dec90 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc48fe00640 10 client.15222 handle_command_reply: tid=0 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc4bbff8740 1 client.15222 shutdown 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 msgr2=0x55d02cc17d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.551+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cc17d90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009230 tx=0x7fc49c009260 comp rx=0 tx=0).stop 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 shutdown_connections 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x55d02c7dec90 0x55d02cc7e670 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d02c84af60 0x55d02cc182d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d02c82a280 0x55d02cc17d90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 --2- 172.21.15.39:0/183341673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d02c8220a0 0x55d02cc17850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 >> 172.21.15.39:0/183341673 conn(0x55d02c83f350 msgr2=0x55d02cbea2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc48fe00640 0 client.15222 ms_handle_reset on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 shutdown_connections 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 1 -- 172.21.15.39:0/183341673 wait complete. 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.552+0000 7fc4bbff8740 20 client.15222 trim_cache size 0 max 16384 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:03:07.557 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24760, 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 90.227207360999998, 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 160.60927803300001 2024-09-21T01:03:07.558 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc4940b6620 msgr2=0x7fc4940b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc4940b6620 0x7fc4940b8b30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0106e60 tx=0x7fc49c005890 comp rx=0 tx=0).stop 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 msgr2=0x7fc4b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 0x7fc4b0075710 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000e9f0 tx=0x7fc4a000eeb0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 shutdown_connections 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4b0109c40 0x7fc4b0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc4940b6620 0x7fc4940b8b30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b01092f0 0x7fc4b0075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 --2- 172.21.15.39:0/688566326 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc4b01039b0 0x7fc4b0075710 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:07.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 >> 172.21.15.39:0/688566326 conn(0x7fc4b00ff400 msgr2=0x7fc4b0101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:07.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 shutdown_connections 2024-09-21T01:03:07.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:07.553+0000 7fc4af200640 1 -- 172.21.15.39:0/688566326 wait complete. 2024-09-21T01:03:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:07 smithi039 ceph-mon[30415]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:07 smithi150 ceph-mon[34945]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:07 smithi160 ceph-mon[34696]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:08.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T01:03:08.929 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:09.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.567+0000 7faf53c00640 1 -- 172.21.15.39:0/1855665586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 msgr2=0x7faf4c10bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.567+0000 7faf53c00640 1 --2- 172.21.15.39:0/1855665586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c10bb00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7faf34009900 tx=0x7faf3402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.567+0000 7faf53c00640 1 -- 172.21.15.39:0/1855665586 shutdown_connections 2024-09-21T01:03:09.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.567+0000 7faf53c00640 1 --2- 172.21.15.39:0/1855665586 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf4c10c040 0x7faf4c112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.568+0000 7faf53c00640 1 --2- 172.21.15.39:0/1855665586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c10bb00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.568+0000 7faf53c00640 1 --2- 172.21.15.39:0/1855665586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 0x7faf4c1061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.568+0000 7faf53c00640 1 -- 172.21.15.39:0/1855665586 >> 172.21.15.39:0/1855665586 conn(0x7faf4c1017e0 msgr2=0x7faf4c103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:09.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.568+0000 7faf53c00640 1 -- 172.21.15.39:0/1855665586 shutdown_connections 2024-09-21T01:03:09.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.568+0000 7faf53c00640 1 -- 172.21.15.39:0/1855665586 wait complete. 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 Processor -- start 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 -- start start 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 0x7faf4c19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf4c10c040 0x7faf4c1a1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c115d00 con 0x7faf4c10b6f0 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c115b80 con 0x7faf4c105db0 2024-09-21T01:03:09.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.569+0000 7faf53c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c115e80 con 0x7faf4c10c040 2024-09-21T01:03:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 0x7faf4c19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57858/0 (socket says 172.21.15.39:57858) 2024-09-21T01:03:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf43e00640 1 -- 172.21.15.39:0/2751542631 learned_addr learned my addr 172.21.15.39:0/2751542631 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf4c10c040 0x7faf4c1a1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51200640 1 -- 172.21.15.39:0/2751542631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf4c10c040 msgr2=0x7faf4c1a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51200640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf4c10c040 0x7faf4c1a1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51200640 1 -- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 msgr2=0x7faf4c19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51200640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf51200640 1 -- 172.21.15.39:0/2751542631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4c1a2700 con 0x7faf4c105db0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.570+0000 7faf43e00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.571+0000 7faf51200640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 0x7faf4c19c790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faf3c00d6d0 tx=0x7faf3c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.571+0000 7faf4a000640 1 -- 172.21.15.39:0/2751542631 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c017070 con 0x7faf4c105db0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.571+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4c1a29f0 con 0x7faf4c105db0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.571+0000 7faf4a000640 1 -- 172.21.15.39:0/2751542631 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3c009820 con 0x7faf4c105db0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.571+0000 7faf4a000640 1 -- 172.21.15.39:0/2751542631 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c004c40 con 0x7faf4c105db0 2024-09-21T01:03:09.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.571+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf4c1a9150 con 0x7faf4c105db0 2024-09-21T01:03:09.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.575+0000 7faf5a87d740 1 Processor -- start 2024-09-21T01:03:09.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.575+0000 7faf5a87d740 1 -- start start 2024-09-21T01:03:09.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf40490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf409d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf40f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf5a87d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5648fceb7d50 con 0x5648fcb79280 2024-09-21T01:03:09.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf5a87d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5648fcea7180 con 0x5648fcb99f60 2024-09-21T01:03:09.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf5a87d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5648fceaaf90 con 0x5648fcb710a0 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf40f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf40f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57862/0 (socket says 172.21.15.39:57862) 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 -- 172.21.15.39:0/3110984173 learned_addr learned my addr 172.21.15.39:0/3110984173 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf51c00640 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf40490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 -- 172.21.15.39:0/3110984173 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 msgr2=0x5648fcf409d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 -- 172.21.15.39:0/3110984173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 msgr2=0x5648fcf40490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf40490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf43e00640 1 -- 172.21.15.39:0/3110984173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648fce71a70 con 0x5648fcb79280 2024-09-21T01:03:09.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.576+0000 7faf51c00640 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf40490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:09.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.577+0000 7faf4a000640 1 -- 172.21.15.39:0/2751542631 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7faf3c005220 con 0x7faf4c105db0 2024-09-21T01:03:09.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.577+0000 7faf43e00640 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf40f10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faf34006010 tx=0x7faf34005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab05509327f3def server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.577+0000 7faf42000640 1 -- 172.21.15.39:0/3110984173 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34035040 con 0x5648fcb79280 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.577+0000 7faf42000640 1 -- 172.21.15.39:0/3110984173 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf34031820 con 0x5648fcb79280 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.577+0000 7faf42000640 1 -- 172.21.15.39:0/3110984173 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340049a0 con 0x5648fcb79280 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 -- 172.21.15.39:0/3110984173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 msgr2=0x5648fcf40f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf40f10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faf34006010 tx=0x7faf34005ed0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 -- 172.21.15.39:0/3110984173 shutdown_connections 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf40f10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3110984173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf40490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 -- 172.21.15.39:0/3110984173 >> 172.21.15.39:0/3110984173 conn(0x5648fcb8e350 msgr2=0x5648fcf3e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 -- 172.21.15.39:0/3110984173 shutdown_connections 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf5a87d740 1 -- 172.21.15.39:0/3110984173 wait complete. 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.578+0000 7faf4a000640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf300b6500 0x7faf300b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.579+0000 7faf4a000640 1 -- 172.21.15.39:0/2751542631 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7faf3c0e74a0 con 0x7faf4c105db0 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.579+0000 7faf43e00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf300b6500 0x7faf300b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.579+0000 7faf5a87d740 1 -- start start 2024-09-21T01:03:09.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.579+0000 7faf5a87d740 1 Processor -- start 2024-09-21T01:03:09.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf43e00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf300b6500 0x7faf300b8a10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faf340057c0 tx=0x7faf34031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:09.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb710a0 0x5648fcf4c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf4c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb99f60 0x5648fcf57f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5648fcd085f0 con 0x5648fcb79280 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5648fce42460 con 0x5648fcb710a0 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf4ec10 con 0x5648fcb99f60 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf5a87d740 10 client.0 _ll_register_callbacks cb 0x5648fce52210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf4c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf51c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb99f60 0x5648fcf57f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf51c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb99f60 0x5648fcf57f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42192/0 (socket says 172.21.15.39:42192) 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf51c00640 1 -- 172.21.15.39:0/630563125 learned_addr learned my addr 172.21.15.39:0/630563125 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.580+0000 7faf51200640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb710a0 0x5648fcf4c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf51200640 1 -- 172.21.15.39:0/630563125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb99f60 msgr2=0x5648fcf57f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf51200640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb99f60 0x5648fcf57f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf51200640 1 -- 172.21.15.39:0/630563125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 msgr2=0x5648fcf4c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf51200640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf4c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf51200640 1 -- 172.21.15.39:0/630563125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648fcc05540 con 0x5648fcb710a0 2024-09-21T01:03:09.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf43e00640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf4c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:09.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf51200640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb710a0 0x5648fcf4c080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0e4950 tx=0x7faf3c0ac7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b0d7662158121c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:09.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf3aa00640 1 -- 172.21.15.39:0/630563125 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c020070 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5648fce44a70 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.581+0000 7faf3aa00640 1 -- 172.21.15.39:0/630563125 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7faf3c0099d0 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf3aa00640 1 -- 172.21.15.39:0/630563125 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf3c0b69f0 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 10 client.25247 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5648fce5f6f0 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf3aa00640 1 -- 172.21.15.39:0/630563125 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5648fce44a70 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf3aa00640 1 -- 172.21.15.39:0/630563125 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faf3c098130 con 0x5648fcb710a0 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 10 client.25247 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:09.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 10 client.25247 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 20 client.25247 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb588e0 0x5648fcf5e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 4 client.25247 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.582+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648fceb9ca0 con 0x5648fcb588e0 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.583+0000 7faf43e00640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb588e0 0x5648fcf5e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.583+0000 7faf43e00640 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb588e0 0x5648fcf5e480 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac1e413d3c33b6a server_cookie=432d9fb6c3ebd9c2 in_seq=0 out_seq=0 2024-09-21T01:03:09.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.583+0000 7faf3aa00640 10 client.25247 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.584+0000 7faf3aa00640 1 -- 172.21.15.39:0/630563125 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5648fceb9ca0 con 0x5648fcb588e0 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.584+0000 7faf3aa00640 10 client.25247 handle_command_reply: tid=0 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.584+0000 7faf5a87d740 1 client.25247 shutdown 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.584+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb710a0 msgr2=0x5648fcf4c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.584+0000 7faf5a87d740 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb710a0 0x5648fcf4c080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0e4950 tx=0x7faf3c0ac7f0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 shutdown_connections 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb588e0 0x5648fcf5e480 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb99f60 0x5648fcf57f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf4c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 --2- 172.21.15.39:0/630563125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb710a0 0x5648fcf4c080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 >> 172.21.15.39:0/630563125 conn(0x5648fcb8e350 msgr2=0x5648fcf43670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:09.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf3aa00640 0 client.25247 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:09.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 shutdown_connections 2024-09-21T01:03:09.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 1 -- 172.21.15.39:0/630563125 wait complete. 2024-09-21T01:03:09.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.585+0000 7faf5a87d740 20 client.25247 trim_cache size 0 max 16384 2024-09-21T01:03:09.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 Processor -- start 2024-09-21T01:03:09.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 -- start start 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf46fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf47520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf4c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf5c7c0 con 0x5648fcb79280 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf5ce90 con 0x5648fcb99f60 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.604+0000 7faf5a87d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf5d560 con 0x5648fcb710a0 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf47520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf47520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57884/0 (socket says 172.21.15.39:57884) 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 -- 172.21.15.39:0/3642776688 learned_addr learned my addr 172.21.15.39:0/3642776688 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:09.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf51c00640 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf4c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf51200640 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf46fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 -- 172.21.15.39:0/3642776688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 msgr2=0x5648fcf46fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf46fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 -- 172.21.15.39:0/3642776688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 msgr2=0x5648fcf4c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf4c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 -- 172.21.15.39:0/3642776688 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648fcf4d140 con 0x5648fcb79280 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.605+0000 7faf43e00640 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf47520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faf34031950 tx=0x7faf34005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4693ea8c17c8bf76 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.606+0000 7faf41600640 1 -- 172.21.15.39:0/3642776688 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34035040 con 0x5648fcb79280 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.606+0000 7faf41600640 1 -- 172.21.15.39:0/3642776688 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf340035f0 con 0x5648fcb79280 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.606+0000 7faf41600640 1 -- 172.21.15.39:0/3642776688 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340646c0 con 0x5648fcb79280 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 -- 172.21.15.39:0/3642776688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 msgr2=0x5648fcf47520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf47520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faf34031950 tx=0x7faf34005ed0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 -- 172.21.15.39:0/3642776688 shutdown_connections 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf4c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf47520 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3642776688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf46fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 -- 172.21.15.39:0/3642776688 >> 172.21.15.39:0/3642776688 conn(0x5648fcb8e350 msgr2=0x5648fcf45920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 -- 172.21.15.39:0/3642776688 shutdown_connections 2024-09-21T01:03:09.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 -- 172.21.15.39:0/3642776688 wait complete. 2024-09-21T01:03:09.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.607+0000 7faf5a87d740 1 -- start start 2024-09-21T01:03:09.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 Processor -- start 2024-09-21T01:03:09.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf3cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf3d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf5e300 con 0x5648fcb79280 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf5dc30 con 0x5648fcb99f60 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5648fcf5e9d0 con 0x5648fcb710a0 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf5a87d740 10 client.0 _ll_register_callbacks cb 0x7faf3c0acf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf3d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf51200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf3cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf51c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf51200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf3cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42212/0 (socket says 172.21.15.39:42212) 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf51200640 1 -- 172.21.15.39:0/3145511233 learned_addr learned my addr 172.21.15.39:0/3145511233 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.608+0000 7faf43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf3d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57892/0 (socket says 172.21.15.39:57892) 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf51200640 1 -- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 msgr2=0x5648fcf656d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf51200640 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf656d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf51200640 1 -- 172.21.15.39:0/3145511233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 msgr2=0x5648fcf3d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf51200640 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf3d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf51200640 1 -- 172.21.15.39:0/3145511233 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648fcf65c10 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf51200640 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf3cd50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0abe40 tx=0x7faf3c0e7380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5f02828fc0d93c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf3aa00640 1 -- 172.21.15.39:0/3145511233 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c020070 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf340035f0 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.609+0000 7faf3aa00640 1 -- 172.21.15.39:0/3145511233 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7faf3c0abec0 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf3aa00640 1 -- 172.21.15.39:0/3145511233 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf3c0a96d0 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 10 client.25246 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5648fcf66e30 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf3aa00640 1 -- 172.21.15.39:0/3145511233 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf340035f0 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf3aa00640 1 -- 172.21.15.39:0/3145511233 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faf3c0a8af0 con 0x5648fcb710a0 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 10 client.25246 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 10 client.25246 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:03:09.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 20 client.25246 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:09.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb2dc90 0x5648fcfcc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:09.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 4 client.25246 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:03:09.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.610+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "status"}) -- 0x5648fceb5750 con 0x5648fcb2dc90 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf43e00640 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb2dc90 0x5648fcfcc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf43e00640 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb2dc90 0x5648fcfcc660 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70144dccc9b01f78 server_cookie=e8a2771961ec3aa2 in_seq=0 out_seq=0 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf3aa00640 10 client.25246 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf3aa00640 1 -- 172.21.15.39:0/3145511233 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5648fceb5750 con 0x5648fcb2dc90 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf3aa00640 10 client.25246 handle_command_reply: tid=0 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf5a87d740 1 client.25246 shutdown 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 msgr2=0x5648fcf3cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.611+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf3cd50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0abe40 tx=0x7faf3c0e7380 comp rx=0 tx=0).stop 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 shutdown_connections 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5648fcb2dc90 0x5648fcfcc660 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648fcb99f60 0x5648fcf656d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648fcb79280 0x5648fcf3d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 --2- 172.21.15.39:0/3145511233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5648fcb710a0 0x5648fcf3cd50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf3aa00640 0 client.25246 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 >> 172.21.15.39:0/3145511233 conn(0x5648fcb8e350 msgr2=0x5648fcf46fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 shutdown_connections 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 1 -- 172.21.15.39:0/3145511233 wait complete. 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf5a87d740 20 client.25246 trim_cache size 0 max 16384 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-09-21T01:03:09.616 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24764, 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 145.11236003600001, 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 160.12662143099999 2024-09-21T01:03:09.617 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf300b6500 msgr2=0x7faf300b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf53c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf300b6500 0x7faf300b8a10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faf340057c0 tx=0x7faf34031400 comp rx=0 tx=0).stop 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 msgr2=0x7faf4c19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.612+0000 7faf53c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 0x7faf4c19c790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faf3c00d6d0 tx=0x7faf3c00db90 comp rx=0 tx=0).stop 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 shutdown_connections 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf4c10c040 0x7faf4c1a1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf4c10b6f0 0x7faf4c19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faf300b6500 0x7faf300b8a10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 --2- 172.21.15.39:0/2751542631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf4c105db0 0x7faf4c19c790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:09.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 >> 172.21.15.39:0/2751542631 conn(0x7faf4c1017e0 msgr2=0x7faf4c10ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 shutdown_connections 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:09.613+0000 7faf53c00640 1 -- 172.21.15.39:0/2751542631 wait complete. 2024-09-21T01:03:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:09 smithi039 ceph-mon[30415]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:09 smithi150 ceph-mon[34945]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:09 smithi160 ceph-mon[34696]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:10.565 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T01:03:10.565 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T01:03:10.581 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:03:10.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-21T01:03:10.966 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:11.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.605+0000 7f647ec00640 1 -- 172.21.15.39:0/3313986929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810be50 msgr2=0x7f6478112680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:11.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.605+0000 7f647ec00640 1 --2- 172.21.15.39:0/3313986929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810be50 0x7f6478112680 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6470009ca0 tx=0x7f647002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 -- 172.21.15.39:0/3313986929 shutdown_connections 2024-09-21T01:03:11.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 --2- 172.21.15.39:0/3313986929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810be50 0x7f6478112680 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 --2- 172.21.15.39:0/3313986929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647810b500 0x7f647810b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 --2- 172.21.15.39:0/3313986929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 0x7f6478105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 -- 172.21.15.39:0/3313986929 >> 172.21.15.39:0/3313986929 conn(0x7f6478101670 msgr2=0x7f6478103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:11.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 -- 172.21.15.39:0/3313986929 shutdown_connections 2024-09-21T01:03:11.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.606+0000 7f647ec00640 1 -- 172.21.15.39:0/3313986929 wait complete. 2024-09-21T01:03:11.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.607+0000 7f647ec00640 1 Processor -- start 2024-09-21T01:03:11.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.607+0000 7f647ec00640 1 -- start start 2024-09-21T01:03:11.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.607+0000 7f647ec00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 0x7f647819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:11.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.607+0000 7f647ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810b500 0x7f647819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:11.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ec00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647810be50 0x7f64781a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:11.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6478115b60 con 0x7f647810b500 2024-09-21T01:03:11.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ec00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64781159e0 con 0x7f6478105bc0 2024-09-21T01:03:11.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ec00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6478115ce0 con 0x7f647810be50 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810b500 0x7f647819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647d800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 0x7f647819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810b500 0x7f647819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57916/0 (socket says 172.21.15.39:57916) 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647ce00640 1 -- 172.21.15.39:0/1939379334 learned_addr learned my addr 172.21.15.39:0/1939379334 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647e200640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647810be50 0x7f64781a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647d800640 1 -- 172.21.15.39:0/1939379334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647810be50 msgr2=0x7f64781a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647d800640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647810be50 0x7f64781a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647d800640 1 -- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810b500 msgr2=0x7f647819ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.608+0000 7f647d800640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810b500 0x7f647819ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f647d800640 1 -- 172.21.15.39:0/1939379334 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64781a2460 con 0x7f6478105bc0 2024-09-21T01:03:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f647d800640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 0x7f647819c4f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e9f0 tx=0x7f646c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c00ccc0 con 0x7f6478105bc0 2024-09-21T01:03:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64781a2750 con 0x7f6478105bc0 2024-09-21T01:03:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f646c0108e0 con 0x7f6478105bc0 2024-09-21T01:03:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c018880 con 0x7f6478105bc0 2024-09-21T01:03:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.609+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64781a8eb0 con 0x7f6478105bc0 2024-09-21T01:03:11.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.610+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6478105fd0 con 0x7f6478105bc0 2024-09-21T01:03:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.613+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f646c010430 con 0x7f6478105bc0 2024-09-21T01:03:11.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.614+0000 7f6476000640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64600b6560 0x7f64600b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.615+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f646c0e7ff0 con 0x7f6478105bc0 2024-09-21T01:03:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.615+0000 7f647ce00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64600b6560 0x7f64600b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.615+0000 7f647ce00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64600b6560 0x7f64600b8a70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6478104ec0 tx=0x7f6464005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.616+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f646c0b27a0 con 0x7f6478105bc0 2024-09-21T01:03:11.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.829+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f6478063800 con 0x7f64600b6560 2024-09-21T01:03:11.830 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:11 smithi039 ceph-mon[30415]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-21T01:03:11.830 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:11 smithi039 ceph-mon[30415]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-09-21T01:03:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.830+0000 7f6476000640 1 -- 172.21.15.39:0/1939379334 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f6478063800 con 0x7f64600b6560 2024-09-21T01:03:11.831 INFO:teuthology.orchestra.run.smithi039.stdout:Backend: cephadm 2024-09-21T01:03:11.831 INFO:teuthology.orchestra.run.smithi039.stdout:Available: Yes 2024-09-21T01:03:11.832 INFO:teuthology.orchestra.run.smithi039.stdout:Paused: No 2024-09-21T01:03:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64600b6560 msgr2=0x7f64600b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64600b6560 0x7f64600b8a70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6478104ec0 tx=0x7f6464005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 msgr2=0x7f647819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 0x7f647819c4f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e9f0 tx=0x7f646c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 shutdown_connections 2024-09-21T01:03:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647810be50 0x7f64781a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647810b500 0x7f647819ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64600b6560 0x7f64600b8a70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 --2- 172.21.15.39:0/1939379334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6478105bc0 0x7f647819c4f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 >> 172.21.15.39:0/1939379334 conn(0x7f6478101670 msgr2=0x7f6478109ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 shutdown_connections 2024-09-21T01:03:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:11.833+0000 7f647ec00640 1 -- 172.21.15.39:0/1939379334 wait complete. 2024-09-21T01:03:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:11 smithi150 ceph-mon[34945]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-21T01:03:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:11 smithi150 ceph-mon[34945]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-09-21T01:03:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:11 smithi160 ceph-mon[34696]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-21T01:03:12.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:11 smithi160 ceph-mon[34696]: mgrmap e22: x(active, since 10m), standbys: z, y 2024-09-21T01:03:12.658 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-21T01:03:13.034 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:13.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 -- 172.21.15.39:0/765252382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 msgr2=0x7fc3e01123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:13.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/765252382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01123c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009ca0 tx=0x7fc3d802a950 comp rx=0 tx=0).stop 2024-09-21T01:03:13.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 -- 172.21.15.39:0/765252382 shutdown_connections 2024-09-21T01:03:13.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/765252382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01123c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/765252382 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3e0102d40 0x7fc3e010fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/765252382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e01023f0 0x7fc3e0102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 -- 172.21.15.39:0/765252382 >> 172.21.15.39:0/765252382 conn(0x7fc3e00fe2d0 msgr2=0x7fc3e0100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:13.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 -- 172.21.15.39:0/765252382 shutdown_connections 2024-09-21T01:03:13.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.702+0000 7fc3e4e00640 1 -- 172.21.15.39:0/765252382 wait complete. 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 Processor -- start 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 -- start start 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3e01023f0 0x7fc3e019c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 0x7fc3e019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0115a90 con 0x7fc3e010ffa0 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0115910 con 0x7fc3e0102d40 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.703+0000 7fc3e4e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0115c10 con 0x7fc3e01023f0 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 0x7fc3e019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:13.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:13.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 0x7fc3e019c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:59762/0 (socket says 172.21.15.39:59762) 2024-09-21T01:03:13.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 -- 172.21.15.39:0/3339869499 learned_addr learned my addr 172.21.15.39:0/3339869499 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:13.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3df400640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3e01023f0 0x7fc3e019c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:13.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 -- 172.21.15.39:0/3339869499 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3e01023f0 msgr2=0x7fc3e019c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3e01023f0 0x7fc3e019c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 -- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 msgr2=0x7fc3e01a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3d7e00640 1 -- 172.21.15.39:0/3339869499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e01a2310 con 0x7fc3e0102d40 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.704+0000 7fc3dfe00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.705+0000 7fc3d7e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 0x7fc3e019c940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800e9f0 tx=0x7fc3c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.705+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c800cd80 con 0x7fc3e0102d40 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.705+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e01a2600 con 0x7fc3e0102d40 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.705+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3c80108e0 con 0x7fc3e0102d40 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.705+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c8018980 con 0x7fc3e0102d40 2024-09-21T01:03:13.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.705+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e01a8d60 con 0x7fc3e0102d40 2024-09-21T01:03:13.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.706+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e0102800 con 0x7fc3e0102d40 2024-09-21T01:03:13.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.708+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7fc3c8010430 con 0x7fc3e0102d40 2024-09-21T01:03:13.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.710+0000 7fc3dcc00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3c40b6560 0x7fc3c40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:13.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.710+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc3c80e7ff0 con 0x7fc3e0102d40 2024-09-21T01:03:13.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.710+0000 7fc3df400640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3c40b6560 0x7fc3c40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:13.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.711+0000 7fc3df400640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3c40b6560 0x7fc3c40b8a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009bd0 tx=0x7fc3d0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:13.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.712+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc3c80b2780 con 0x7fc3e0102d40 2024-09-21T01:03:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:13 smithi039 ceph-mon[30415]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:13 smithi039 ceph-mon[30415]: pgmap v380: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T01:03:13.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.920+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fc3e0112100 con 0x7fc3c40b6560 2024-09-21T01:03:13.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.933+0000 7fc3dcc00640 1 -- 172.21.15.39:0/3339869499 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fc3e0112100 con 0x7fc3c40b6560 2024-09-21T01:03:13.934 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:03:13.934 INFO:teuthology.orchestra.run.smithi039.stdout:mds.a smithi039 running (2m) 22s ago 2m 85.3M - 19.3.0-5120-g18432218 ddce307e1f32 48088db5c710 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.b smithi150 running (2m) 30s ago 2m 94.3M - 19.3.0-5120-g18432218 ddce307e1f32 559a4fc2128e 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.c smithi160 running (3m) 30s ago 2m 86.4M - 19.3.0-5120-g18432218 ddce307e1f32 a5b3475708b0 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.d smithi039 running (3m) 22s ago 3m 80.9M - 19.3.0-5120-g18432218 ddce307e1f32 7e6a80933bc9 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.e smithi150 running (3m) 30s ago 3m 83.3M - 19.3.0-5120-g18432218 ddce307e1f32 11c0020994b7 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.f smithi160 running (3m) 30s ago 3m 87.8M - 19.3.0-5120-g18432218 ddce307e1f32 a0e492f5e619 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.g smithi039 running (2m) 22s ago 2m 80.7M - 19.3.0-5120-g18432218 ddce307e1f32 2b422ce1ca6a 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.h smithi150 running (2m) 30s ago 2m 85.7M - 19.3.0-5120-g18432218 ddce307e1f32 2e6bdd1c2280 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.i smithi160 running (2m) 30s ago 2m 86.1M - 19.3.0-5120-g18432218 ddce307e1f32 f5da40691c52 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.j smithi039 running (3m) 22s ago 3m 80.9M - 19.3.0-5120-g18432218 ddce307e1f32 5c49624a179c 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.k smithi150 running (3m) 30s ago 3m 82.3M - 19.3.0-5120-g18432218 ddce307e1f32 b1f17fc13254 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mds.l smithi160 running (3m) 30s ago 3m 82.6M - 19.3.0-5120-g18432218 ddce307e1f32 c202ad70b511 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.x smithi039 *:9283,8765 running (12m) 22s ago 12m 586M - 19.3.0-5120-g18432218 ddce307e1f32 9dd56f0e6631 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.y smithi150 *:8443,8765 running (9m) 30s ago 9m 525M - 19.3.0-5120-g18432218 ddce307e1f32 9814ec731fc1 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.z smithi160 *:8443,8765 running (9m) 30s ago 9m 525M - 19.3.0-5120-g18432218 ddce307e1f32 143afbe3a556 2024-09-21T01:03:13.935 INFO:teuthology.orchestra.run.smithi039.stdout:mon.a smithi039 running (12m) 22s ago 12m 120M 2048M 19.3.0-5120-g18432218 ddce307e1f32 74510f689f9d 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:mon.b smithi150 running (10m) 30s ago 10m 111M 2048M 19.3.0-5120-g18432218 ddce307e1f32 39ea9691389c 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:mon.c smithi160 running (10m) 30s ago 10m 112M 2048M 19.3.0-5120-g18432218 ddce307e1f32 5c44cd059d0e 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (8m) 22s ago 8m 112M 4272M 19.3.0-5120-g18432218 ddce307e1f32 3927d12efc37 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi150 running (8m) 30s ago 8m 114M 4272M 19.3.0-5120-g18432218 ddce307e1f32 69d8995f2833 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi160 running (8m) 30s ago 8m 118M 4272M 19.3.0-5120-g18432218 ddce307e1f32 ec22a90618a2 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi039 running (7m) 22s ago 7m 119M 4272M 19.3.0-5120-g18432218 ddce307e1f32 998a0e3dbd36 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi150 running (6m) 30s ago 6m 118M 4272M 19.3.0-5120-g18432218 ddce307e1f32 66df1b80b32c 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi160 running (6m) 30s ago 6m 121M 4272M 19.3.0-5120-g18432218 ddce307e1f32 20546f6c12c1 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.6 smithi039 running (5m) 22s ago 5m 117M 4272M 19.3.0-5120-g18432218 ddce307e1f32 60f525769fef 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.7 smithi150 running (5m) 30s ago 5m 119M 4272M 19.3.0-5120-g18432218 ddce307e1f32 1ad3a6744966 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.8 smithi160 running (5m) 30s ago 5m 111M 4272M 19.3.0-5120-g18432218 ddce307e1f32 bd24e72e0145 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.9 smithi039 running (4m) 22s ago 4m 109M 4272M 19.3.0-5120-g18432218 ddce307e1f32 2dda4214b923 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.10 smithi150 running (4m) 30s ago 4m 114M 4272M 19.3.0-5120-g18432218 ddce307e1f32 c21126c06644 2024-09-21T01:03:13.936 INFO:teuthology.orchestra.run.smithi039.stdout:osd.11 smithi160 running (3m) 30s ago 3m 109M 4272M 19.3.0-5120-g18432218 ddce307e1f32 2d5f5901e912 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3c40b6560 msgr2=0x7fc3c40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3c40b6560 0x7fc3c40b8a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009bd0 tx=0x7fc3d0005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 msgr2=0x7fc3e019c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 0x7fc3e019c940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800e9f0 tx=0x7fc3c800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 shutdown_connections 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3e010ffa0 0x7fc3e01a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3c40b6560 0x7fc3c40b8a70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3e0102d40 0x7fc3e019c940 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 --2- 172.21.15.39:0/3339869499 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3e01023f0 0x7fc3e019c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:13.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.935+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 >> 172.21.15.39:0/3339869499 conn(0x7fc3e00fe2d0 msgr2=0x7fc3e01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:13.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.936+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 shutdown_connections 2024-09-21T01:03:13.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:13.936+0000 7fc3e4e00640 1 -- 172.21.15.39:0/3339869499 wait complete. 2024-09-21T01:03:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:13 smithi150 ceph-mon[34945]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:13 smithi150 ceph-mon[34945]: pgmap v380: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T01:03:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:13 smithi160 ceph-mon[34696]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:14.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:13 smithi160 ceph-mon[34696]: pgmap v380: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T01:03:14.789 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-21T01:03:15.168 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:15.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 -- 172.21.15.39:0/2350709579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8103990 msgr2=0x7f4bc8103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 --2- 172.21.15.39:0/2350709579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8103990 0x7f4bc8103da0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009900 tx=0x7f4bbc02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 -- 172.21.15.39:0/2350709579 shutdown_connections 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 --2- 172.21.15.39:0/2350709579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8109c20 0x7f4bc8110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 --2- 172.21.15.39:0/2350709579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bc81092d0 0x7f4bc81096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 --2- 172.21.15.39:0/2350709579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8103990 0x7f4bc8103da0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 -- 172.21.15.39:0/2350709579 >> 172.21.15.39:0/2350709579 conn(0x7f4bc80ff440 msgr2=0x7f4bc8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 -- 172.21.15.39:0/2350709579 shutdown_connections 2024-09-21T01:03:15.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.845+0000 7f4bce400640 1 -- 172.21.15.39:0/2350709579 wait complete. 2024-09-21T01:03:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.846+0000 7f4bce400640 1 Processor -- start 2024-09-21T01:03:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 -- start start 2024-09-21T01:03:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8103990 0x7f4bc819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bc81092d0 0x7f4bc819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8109c20 0x7f4bc819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc8113930 con 0x7f4bc8109c20 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc81137b0 con 0x7f4bc81092d0 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bce400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc8113ab0 con 0x7f4bc8103990 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bcda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8109c20 0x7f4bc819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bcda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8109c20 0x7f4bc819fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58212/0 (socket says 172.21.15.39:58212) 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bcd000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8103990 0x7f4bc819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.847+0000 7f4bcda00640 1 -- 172.21.15.39:0/2826164222 learned_addr learned my addr 172.21.15.39:0/2826164222 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bc7e00640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bc81092d0 0x7f4bc819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bcd000640 1 -- 172.21.15.39:0/2826164222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bc81092d0 msgr2=0x7f4bc819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bcd000640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bc81092d0 0x7f4bc819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bcd000640 1 -- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8109c20 msgr2=0x7f4bc819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bcd000640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8109c20 0x7f4bc819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bcd000640 1 -- 172.21.15.39:0/2826164222 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc81a0270 con 0x7f4bc8103990 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.848+0000 7f4bcd000640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8103990 0x7f4bc819a300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc02a9e0 tx=0x7f4bbc003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:15.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.849+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc03f070 con 0x7f4bc8103990 2024-09-21T01:03:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.849+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bbc031d60 con 0x7f4bc8103990 2024-09-21T01:03:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.849+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc81a0500 con 0x7f4bc8103990 2024-09-21T01:03:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.849+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc035030 con 0x7f4bc8103990 2024-09-21T01:03:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.849+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc81a6cc0 con 0x7f4bc8103990 2024-09-21T01:03:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.850+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc8104c10 con 0x7f4bc8103990 2024-09-21T01:03:15.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.852+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f4bbc042070 con 0x7f4bc8103990 2024-09-21T01:03:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.853+0000 7f4bc5600640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4bb00b6560 0x7f4bb00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.854+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4bbc108500 con 0x7f4bc8103990 2024-09-21T01:03:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.854+0000 7f4bc7e00640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4bb00b6560 0x7f4bb00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.854+0000 7f4bc7e00640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4bb00b6560 0x7f4bb00b8a70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8109b90 tx=0x7f4bb400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:15.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:15.856+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4bbc0d2c70 con 0x7f4bc8103990 2024-09-21T01:03:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:15 smithi150 ceph-mon[34945]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:15 smithi150 ceph-mon[34945]: pgmap v381: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-21T01:03:16.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.066+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f4bc810ba50 con 0x7f4bb00b6560 2024-09-21T01:03:16.068 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:15 smithi039 ceph-mon[30415]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:16.068 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:15 smithi039 ceph-mon[30415]: pgmap v381: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-21T01:03:16.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.074+0000 7f4bc5600640 1 -- 172.21.15.39:0/2826164222 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f4bc810ba50 con 0x7f4bb00b6560 2024-09-21T01:03:16.074 INFO:teuthology.orchestra.run.smithi039.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-21T01:03:16.074 INFO:teuthology.orchestra.run.smithi039.stdout:mds.all 12/12 33s ago 3m smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi160=c;smithi160=f;smithi160=i;smithi160=l;count:12 2024-09-21T01:03:16.075 INFO:teuthology.orchestra.run.smithi039.stdout:mgr 3/3 33s ago 10m smithi039=x;smithi150=y;smithi160=z;count:3 2024-09-21T01:03:16.075 INFO:teuthology.orchestra.run.smithi039.stdout:mon 3/3 33s ago 10m smithi039:172.21.15.39=a;smithi150:172.21.15.150=b;smithi160:172.21.15.160=c;count:3 2024-09-21T01:03:16.075 INFO:teuthology.orchestra.run.smithi039.stdout:osd 12 33s ago - 2024-09-21T01:03:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4bb00b6560 msgr2=0x7f4bb00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4bb00b6560 0x7f4bb00b8a70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8109b90 tx=0x7f4bb400b410 comp rx=0 tx=0).stop 2024-09-21T01:03:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8103990 msgr2=0x7f4bc819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8103990 0x7f4bc819a300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc02a9e0 tx=0x7f4bbc003cd0 comp rx=0 tx=0).stop 2024-09-21T01:03:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 shutdown_connections 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bc8109c20 0x7f4bc819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4bb00b6560 0x7f4bb00b8a70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bc81092d0 0x7f4bc819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 --2- 172.21.15.39:0/2826164222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4bc8103990 0x7f4bc819a300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 >> 172.21.15.39:0/2826164222 conn(0x7f4bc80ff440 msgr2=0x7f4bc8101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 shutdown_connections 2024-09-21T01:03:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:16.076+0000 7f4bce400640 1 -- 172.21.15.39:0/2826164222 wait complete. 2024-09-21T01:03:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:15 smithi160 ceph-mon[34696]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:15 smithi160 ceph-mon[34696]: pgmap v381: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-09-21T01:03:17.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-21T01:03:17.399 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:17 smithi039 ceph-mon[30415]: pgmap v382: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:17 smithi039 ceph-mon[30415]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:17 smithi150 ceph-mon[34945]: pgmap v382: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:17 smithi150 ceph-mon[34945]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 -- 172.21.15.39:0/920155445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810be40 msgr2=0x7ff548112670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 --2- 172.21.15.39:0/920155445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810be40 0x7ff548112670 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff540009ca0 tx=0x7ff54002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 -- 172.21.15.39:0/920155445 shutdown_connections 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 --2- 172.21.15.39:0/920155445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810be40 0x7ff548112670 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 --2- 172.21.15.39:0/920155445 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff54810b4f0 0x7ff54810b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 --2- 172.21.15.39:0/920155445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff548105bb0 0x7ff548105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 -- 172.21.15.39:0/920155445 >> 172.21.15.39:0/920155445 conn(0x7ff548101680 msgr2=0x7ff548103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:18.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 -- 172.21.15.39:0/920155445 shutdown_connections 2024-09-21T01:03:18.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.104+0000 7ff54e400640 1 -- 172.21.15.39:0/920155445 wait complete. 2024-09-21T01:03:18.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.105+0000 7ff54e400640 1 Processor -- start 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 -- start start 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff548105bb0 0x7ff54819c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 0x7ff54819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff54810be40 0x7ff5481a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff548115b00 con 0x7ff54810b4f0 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff548115980 con 0x7ff548105bb0 2024-09-21T01:03:18.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54e400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff548115c80 con 0x7ff54810be40 2024-09-21T01:03:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:17 smithi160 ceph-mon[34696]: pgmap v382: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:17 smithi160 ceph-mon[34696]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:18.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff547e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 0x7ff54819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:18.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff547e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 0x7ff54819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58234/0 (socket says 172.21.15.39:58234) 2024-09-21T01:03:18.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff547e00640 1 -- 172.21.15.39:0/3776519712 learned_addr learned my addr 172.21.15.39:0/3776519712 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:18.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.106+0000 7ff54d000640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff548105bb0 0x7ff54819c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff54da00640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff54810be40 0x7ff5481a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff547e00640 1 -- 172.21.15.39:0/3776519712 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff54810be40 msgr2=0x7ff5481a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff547e00640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff54810be40 0x7ff5481a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff547e00640 1 -- 172.21.15.39:0/3776519712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff548105bb0 msgr2=0x7ff54819c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff547e00640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff548105bb0 0x7ff54819c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff547e00640 1 -- 172.21.15.39:0/3776519712 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5481a2420 con 0x7ff54810b4f0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.107+0000 7ff547e00640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 0x7ff54819ca50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff53400e9c0 tx=0x7ff53400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.108+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53400ccc0 con 0x7ff54810b4f0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.108+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5340054d0 con 0x7ff54810b4f0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.108+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534010430 con 0x7ff54810b4f0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.108+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5481a2710 con 0x7ff54810b4f0 2024-09-21T01:03:18.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.108+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5481a8e70 con 0x7ff54810b4f0 2024-09-21T01:03:18.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.109+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff548105fc0 con 0x7ff54810b4f0 2024-09-21T01:03:18.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.111+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7ff534005020 con 0x7ff54810b4f0 2024-09-21T01:03:18.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.113+0000 7ff545600640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff5300b65e0 0x7ff5300b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:18.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.113+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff5340e7b50 con 0x7ff54810b4f0 2024-09-21T01:03:18.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.113+0000 7ff54d000640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff5300b65e0 0x7ff5300b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:18.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.114+0000 7ff54d000640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff5300b65e0 0x7ff5300b8af0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0095f0 tx=0x7ff53c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:18.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.117+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff5340adee0 con 0x7ff54810b4f0 2024-09-21T01:03:18.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.323+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7ff548063800 con 0x7ff5300b65e0 2024-09-21T01:03:18.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.324+0000 7ff545600640 1 -- 172.21.15.39:0/3776519712 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7ff548063800 con 0x7ff5300b65e0 2024-09-21T01:03:18.325 INFO:teuthology.orchestra.run.smithi039.stdout:HOST ADDR LABELS STATUS 2024-09-21T01:03:18.325 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039 172.21.15.39 2024-09-21T01:03:18.325 INFO:teuthology.orchestra.run.smithi039.stdout:smithi150 172.21.15.150 2024-09-21T01:03:18.325 INFO:teuthology.orchestra.run.smithi039.stdout:smithi160 172.21.15.160 2024-09-21T01:03:18.325 INFO:teuthology.orchestra.run.smithi039.stdout:3 hosts in cluster 2024-09-21T01:03:18.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff5300b65e0 msgr2=0x7ff5300b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:18.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff5300b65e0 0x7ff5300b8af0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0095f0 tx=0x7ff53c005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:18.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 msgr2=0x7ff54819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:18.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 0x7ff54819ca50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff53400e9c0 tx=0x7ff53400ee80 comp rx=0 tx=0).stop 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 shutdown_connections 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff54810be40 0x7ff5481a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54810b4f0 0x7ff54819ca50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff5300b65e0 0x7ff5300b8af0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 --2- 172.21.15.39:0/3776519712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff548105bb0 0x7ff54819c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 >> 172.21.15.39:0/3776519712 conn(0x7ff548101680 msgr2=0x7ff54810abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.327+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 shutdown_connections 2024-09-21T01:03:18.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:18.328+0000 7ff54e400640 1 -- 172.21.15.39:0/3776519712 wait complete. 2024-09-21T01:03:19.172 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-21T01:03:19.552 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:20.028 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:19 smithi039 ceph-mon[30415]: pgmap v383: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:20.130 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:19 smithi039 ceph-mon[30415]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:20.130 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:19 smithi160 ceph-mon[34696]: pgmap v383: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:20.131 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:19 smithi160 ceph-mon[34696]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:20.131 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:19 smithi150 ceph-mon[34945]: pgmap v383: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:20.131 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:19 smithi150 ceph-mon[34945]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 -- 172.21.15.39:0/129715630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 msgr2=0x7f64b410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 --2- 172.21.15.39:0/129715630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 0x7f64b410b910 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f649c009900 tx=0x7f649c02a410 comp rx=0 tx=0).stop 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 -- 172.21.15.39:0/129715630 shutdown_connections 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 --2- 172.21.15.39:0/129715630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b410be50 0x7f64b4112680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 --2- 172.21.15.39:0/129715630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 0x7f64b410b910 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 --2- 172.21.15.39:0/129715630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64b4105bc0 0x7f64b4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 -- 172.21.15.39:0/129715630 >> 172.21.15.39:0/129715630 conn(0x7f64b4101670 msgr2=0x7f64b4103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 -- 172.21.15.39:0/129715630 shutdown_connections 2024-09-21T01:03:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.208+0000 7f64b3200640 1 -- 172.21.15.39:0/129715630 wait complete. 2024-09-21T01:03:20.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.209+0000 7f64b3200640 1 Processor -- start 2024-09-21T01:03:20.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.209+0000 7f64b3200640 1 -- start start 2024-09-21T01:03:20.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b3200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 0x7f64b419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:20.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 0x7f64b419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:20.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b3200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64b410be50 0x7f64b41a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:20.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4115b10 con 0x7f64b410b500 2024-09-21T01:03:20.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b3200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4115990 con 0x7f64b410be50 2024-09-21T01:03:20.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b3200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4115c90 con 0x7f64b4105bc0 2024-09-21T01:03:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 0x7f64b419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64abe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 0x7f64b419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 0x7f64b419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34446/0 (socket says 172.21.15.39:34446) 2024-09-21T01:03:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b1e00640 1 -- 172.21.15.39:0/3434757972 learned_addr learned my addr 172.21.15.39:0/3434757972 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.210+0000 7f64b2800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64b410be50 0x7f64b41a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b1e00640 1 -- 172.21.15.39:0/3434757972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64b410be50 msgr2=0x7f64b41a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b1e00640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64b410be50 0x7f64b41a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b1e00640 1 -- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 msgr2=0x7f64b419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b1e00640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 0x7f64b419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b1e00640 1 -- 172.21.15.39:0/3434757972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b41a2450 con 0x7f64b4105bc0 2024-09-21T01:03:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b1e00640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 0x7f64b419c4e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f64a400e9f0 tx=0x7f64a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a400cd80 con 0x7f64b4105bc0 2024-09-21T01:03:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b41a2740 con 0x7f64b4105bc0 2024-09-21T01:03:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a40108e0 con 0x7f64b4105bc0 2024-09-21T01:03:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.211+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4018980 con 0x7f64b4105bc0 2024-09-21T01:03:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.212+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b41a8ea0 con 0x7f64b4105bc0 2024-09-21T01:03:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.213+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b4105fd0 con 0x7f64b4105bc0 2024-09-21T01:03:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.215+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f64a4010430 con 0x7f64b4105bc0 2024-09-21T01:03:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.217+0000 7f64aaa00640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64980b6560 0x7f64980b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.217+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f64a40e7fc0 con 0x7f64b4105bc0 2024-09-21T01:03:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.217+0000 7f64abe00640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64980b6560 0x7f64980b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.218+0000 7f64abe00640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64980b6560 0x7f64980b8a70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f649c009230 tx=0x7f649c031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:20.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.219+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f64a40b2790 con 0x7f64b4105bc0 2024-09-21T01:03:20.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.429+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f64b4063800 con 0x7f64980b6560 2024-09-21T01:03:20.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.431+0000 7f64aaa00640 1 -- 172.21.15.39:0/3434757972 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f64b4063800 con 0x7f64980b6560 2024-09-21T01:03:20.431 INFO:teuthology.orchestra.run.smithi039.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-21T01:03:20.432 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102QX4P0KGN 3726G No 4m ago Has a FileSystem, LVM detected 2024-09-21T01:03:20.432 INFO:teuthology.orchestra.run.smithi039.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-09-21T01:03:20.432 INFO:teuthology.orchestra.run.smithi039.stdout:smithi160 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JY400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-21T01:03:20.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64980b6560 msgr2=0x7f64980b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:20.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64980b6560 0x7f64980b8a70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f649c009230 tx=0x7f649c031400 comp rx=0 tx=0).stop 2024-09-21T01:03:20.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 msgr2=0x7f64b419c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:20.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 0x7f64b419c4e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f64a400e9f0 tx=0x7f64a400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 shutdown_connections 2024-09-21T01:03:20.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64980b6560 0x7f64980b8a70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64b410be50 0x7f64b41a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b410b500 0x7f64b419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 --2- 172.21.15.39:0/3434757972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f64b4105bc0 0x7f64b419c4e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 >> 172.21.15.39:0/3434757972 conn(0x7f64b4101670 msgr2=0x7f64b410abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 shutdown_connections 2024-09-21T01:03:20.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:20.434+0000 7f64b3200640 1 -- 172.21.15.39:0/3434757972 wait complete. 2024-09-21T01:03:21.288 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T01:03:21.297 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:03:21.297 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph fs dump' 2024-09-21T01:03:21.691 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:21.997 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:21 smithi039 ceph-mon[30415]: pgmap v384: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:21.998 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:21 smithi039 ceph-mon[30415]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:21 smithi150 ceph-mon[34945]: pgmap v384: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:21 smithi150 ceph-mon[34945]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:21 smithi160 ceph-mon[34696]: pgmap v384: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-09-21T01:03:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:21 smithi160 ceph-mon[34696]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:22.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 -- 172.21.15.39:0/402934096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93541047c0 msgr2=0x7f9354104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 --2- 172.21.15.39:0/402934096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93541047c0 0x7f9354104bd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9344009900 tx=0x7f934402a410 comp rx=0 tx=0).stop 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 -- 172.21.15.39:0/402934096 shutdown_connections 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 --2- 172.21.15.39:0/402934096 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f935410dc20 0x7f9354110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 --2- 172.21.15.39:0/402934096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9354105110 0x7f935410d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 --2- 172.21.15.39:0/402934096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93541047c0 0x7f9354104bd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 -- 172.21.15.39:0/402934096 >> 172.21.15.39:0/402934096 conn(0x7f93540fe2c0 msgr2=0x7f9354100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 -- 172.21.15.39:0/402934096 shutdown_connections 2024-09-21T01:03:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.369+0000 7f9359c00640 1 -- 172.21.15.39:0/402934096 wait complete. 2024-09-21T01:03:22.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.370+0000 7f9359c00640 1 Processor -- start 2024-09-21T01:03:22.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 -- start start 2024-09-21T01:03:22.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 0x7f9354102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:22.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9354105110 0x7f9354102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:22.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f935410dc20 0x7f9354102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:22.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f93541138d0 con 0x7f935410dc20 2024-09-21T01:03:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9354113750 con 0x7f9354105110 2024-09-21T01:03:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.371+0000 7f9359c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9354113a50 con 0x7f93541047c0 2024-09-21T01:03:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 0x7f9354102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 0x7f9354102500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34470/0 (socket says 172.21.15.39:34470) 2024-09-21T01:03:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 -- 172.21.15.39:0/1451353488 learned_addr learned my addr 172.21.15.39:0/1451353488 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f934be00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9354105110 0x7f9354102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9359200640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f935410dc20 0x7f9354102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 -- 172.21.15.39:0/1451353488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9354105110 msgr2=0x7f9354102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9354105110 0x7f9354102a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 -- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f935410dc20 msgr2=0x7f9354102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f935410dc20 0x7f9354102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.372+0000 7f9353e00640 1 -- 172.21.15.39:0/1451353488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93541a8600 con 0x7f93541047c0 2024-09-21T01:03:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.373+0000 7f9353e00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 0x7f9354102500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9344005ab0 tx=0x7f9344003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.373+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934403f070 con 0x7f93541047c0 2024-09-21T01:03:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.373+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9344031d60 con 0x7f93541047c0 2024-09-21T01:03:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.373+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93541a8890 con 0x7f93541047c0 2024-09-21T01:03:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.373+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344035030 con 0x7f93541047c0 2024-09-21T01:03:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.373+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93541a8d70 con 0x7f93541047c0 2024-09-21T01:03:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.374+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9354104bd0 con 0x7f93541047c0 2024-09-21T01:03:22.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.377+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f9344042070 con 0x7f93541047c0 2024-09-21T01:03:22.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.378+0000 7f9351600640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f93380b6560 0x7f93380b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:22.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.378+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f93441086c0 con 0x7f93541047c0 2024-09-21T01:03:22.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.379+0000 7f934be00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f93380b6560 0x7f93380b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:22.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.379+0000 7f934be00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f93380b6560 0x7f93380b8a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9354110100 tx=0x7f933c00b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:22.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.380+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f93440d2e30 con 0x7f93541047c0 2024-09-21T01:03:22.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.621+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f93541a90e0 con 0x7f93541047c0 2024-09-21T01:03:22.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.622+0000 7f9351600640 1 -- 172.21.15.39:0/1451353488 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3088 (secure 0 0 0) 0x7f93440d7030 con 0x7f93541047c0 2024-09-21T01:03:22.622 INFO:teuthology.orchestra.run.smithi039.stdout:e26 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:btime 2024-09-21T01:01:44:370343+0000 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 26 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-09-21T01:00:43.758917+0000 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-09-21T01:01:43.492789+0000 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-09-21T01:03:22.623 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 300 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 0 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 3 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:in 0,1,2 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24764,1=24760,2=24742} 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-09-21T01:03:22.624 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3,4] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data disabled 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 0 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:qdb_cluster leader: 24764 members: 24764,24760,24742 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b{0:24764} state up:active seq 5 addr [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c{0:24727} state up:standby-replay seq 1 addr [v2:172.21.15.160:6836/1965195431,v1:172.21.15.160:6837/1965195431] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a{1:24760} state up:active seq 19 addr [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.h{1:24725} state up:standby-replay seq 1 addr [v2:172.21.15.150:6836/555411539,v1:172.21.15.150:6837/555411539] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.i{2:24742} state up:active seq 21 addr [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.f{2:24715} state up:standby-replay seq 1 addr [v2:172.21.15.160:6834/2680876155,v1:172.21.15.160:6835/2680876155] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.j{-1:14667} state up:standby seq 1 addr [v2:172.21.15.39:6834/319446739,v1:172.21.15.39:6835/319446739] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.g{-1:14733} state up:standby seq 1 addr [v2:172.21.15.39:6838/443106775,v1:172.21.15.39:6839/443106775] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.k{-1:24671} state up:standby seq 1 addr [v2:172.21.15.150:6832/845239112,v1:172.21.15.150:6833/845239112] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.l{-1:24673} state up:standby seq 1 addr [v2:172.21.15.160:6832/166386963,v1:172.21.15.160:6833/166386963] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.e{-1:24701} state up:standby seq 1 addr [v2:172.21.15.150:6834/2170415276,v1:172.21.15.150:6835/2170415276] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d{-1:24703} state up:standby seq 1 addr [v2:172.21.15.39:6836/1544048762,v1:172.21.15.39:6837/1544048762] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.624+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f93380b6560 msgr2=0x7f93380b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.624+0000 7f9359c00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f93380b6560 0x7f93380b8a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9354110100 tx=0x7f933c00b410 comp rx=0 tx=0).stop 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.624+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 msgr2=0x7f9354102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.624+0000 7f9359c00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 0x7f9354102500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9344005ab0 tx=0x7f9344003cd0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 shutdown_connections 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f935410dc20 0x7f9354102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f93380b6560 0x7f93380b8a70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9354105110 0x7f9354102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 --2- 172.21.15.39:0/1451353488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f93541047c0 0x7f9354102500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 >> 172.21.15.39:0/1451353488 conn(0x7f93540fe2c0 msgr2=0x7f9354107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 shutdown_connections 2024-09-21T01:03:22.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:22.625+0000 7f9359c00640 1 -- 172.21.15.39:0/1451353488 wait complete. 2024-09-21T01:03:22.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:22 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1451353488' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T01:03:23.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:22 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1451353488' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T01:03:23.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:22 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1451353488' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T01:03:23.782 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- bash -c 'ceph osd dump' 2024-09-21T01:03:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:23 smithi150 ceph-mon[34945]: pgmap v385: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T01:03:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:23 smithi160 ceph-mon[34696]: pgmap v385: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T01:03:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:23 smithi039 ceph-mon[30415]: pgmap v385: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T01:03:24.169 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:24.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 -- 172.21.15.39:0/4275421166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d01100e0 msgr2=0x7ff6d0112520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:24.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 --2- 172.21.15.39:0/4275421166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d01100e0 0x7ff6d0112520 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009be0 tx=0x7ff6c802a6c0 comp rx=0 tx=0).stop 2024-09-21T01:03:24.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 -- 172.21.15.39:0/4275421166 shutdown_connections 2024-09-21T01:03:24.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 --2- 172.21.15.39:0/4275421166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d01100e0 0x7ff6d0112520 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:24.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 --2- 172.21.15.39:0/4275421166 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6d0076100 0x7ff6d010fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:24.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 --2- 172.21.15.39:0/4275421166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d00757b0 0x7ff6d0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:24.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 -- 172.21.15.39:0/4275421166 >> 172.21.15.39:0/4275421166 conn(0x7ff6d00ff5d0 msgr2=0x7ff6d0101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:24.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 -- 172.21.15.39:0/4275421166 shutdown_connections 2024-09-21T01:03:24.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.799+0000 7ff6d6400640 1 -- 172.21.15.39:0/4275421166 wait complete. 2024-09-21T01:03:24.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 Processor -- start 2024-09-21T01:03:24.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 -- start start 2024-09-21T01:03:24.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 0x7ff6d019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:24.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d0076100 0x7ff6d019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:24.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6d01100e0 0x7ff6d01a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0115c90 con 0x7ff6d00757b0 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0115b10 con 0x7ff6d0076100 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6d6400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0115e10 con 0x7ff6d01100e0 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 0x7ff6d019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.801+0000 7ff6cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 0x7ff6d019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37200/0 (socket says 172.21.15.39:37200) 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 -- 172.21.15.39:0/2817781043 learned_addr learned my addr 172.21.15.39:0/2817781043 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cfe00640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6d01100e0 0x7ff6d01a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6c7e00640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d0076100 0x7ff6d019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 -- 172.21.15.39:0/2817781043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6d01100e0 msgr2=0x7ff6d01a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6d01100e0 0x7ff6d01a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 -- 172.21.15.39:0/2817781043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d0076100 msgr2=0x7ff6d019cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d0076100 0x7ff6d019cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 -- 172.21.15.39:0/2817781043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d01a2630 con 0x7ff6d00757b0 2024-09-21T01:03:24.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6c7e00640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d0076100 0x7ff6d019cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:03:24.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6cf400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 0x7ff6d019c6c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000ea60 tx=0x7ff6c000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:24.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.802+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c000cde0 con 0x7ff6d00757b0 2024-09-21T01:03:24.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.803+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6c00035f0 con 0x7ff6d00757b0 2024-09-21T01:03:24.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.803+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0010690 con 0x7ff6d00757b0 2024-09-21T01:03:24.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.803+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d01a2920 con 0x7ff6d00757b0 2024-09-21T01:03:24.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.803+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d01a9080 con 0x7ff6d00757b0 2024-09-21T01:03:24.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.804+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d0075bc0 con 0x7ff6d00757b0 2024-09-21T01:03:24.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.806+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7ff6c00037a0 con 0x7ff6d00757b0 2024-09-21T01:03:24.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.809+0000 7ff6ccc00640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6b40b65e0 0x7ff6b40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:24.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.809+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff6c00e7d60 con 0x7ff6d00757b0 2024-09-21T01:03:24.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.809+0000 7ff6c7e00640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6b40b65e0 0x7ff6b40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:24.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.810+0000 7ff6c7e00640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6b40b65e0 0x7ff6b40b8af0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff6d01123b0 tx=0x7ff6b8009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:24.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:24.811+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff6c0056710 con 0x7ff6d00757b0 2024-09-21T01:03:25.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.014+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7ff6d01a9430 con 0x7ff6d00757b0 2024-09-21T01:03:25.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.015+0000 7ff6ccc00640 1 -- 172.21.15.39:0/2817781043 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+6096 (secure 0 0 0) 0x7ff6c00ae0f0 con 0x7ff6d00757b0 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 90 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:fsid 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-09-21T00:51:02.311192+0000 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-09-21T01:02:10.480352+0000 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:crush_version 27 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:full_ratio 0.95 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:backfillfull_ratio 0.9 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:nearfull_ratio 0.85 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:require_min_compat_client luminous 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:min_compat_client luminous 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:require_osd_release squid 2024-09-21T01:03:25.016 INFO:teuthology.orchestra.run.smithi039.stdout:stretch_mode_enabled false 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.69 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:max_osd 12 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 up in weight 1 up_from 9 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6802/1725836474,v1:172.21.15.39:6803/1725836474] [v2:172.21.15.39:6804/1725836474,v1:172.21.15.39:6805/1725836474] exists,up db1d8552-f65b-468d-b4f6-a019f730b0fd 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 up in weight 1 up_from 14 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6800/4107270363,v1:172.21.15.150:6801/4107270363] [v2:172.21.15.150:6802/4107270363,v1:172.21.15.150:6803/4107270363] exists,up 4a09b890-68cb-4443-b9eb-f1bf8557126e 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 up in weight 1 up_from 19 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6800/4131806045,v1:172.21.15.160:6801/4131806045] [v2:172.21.15.160:6802/4131806045,v1:172.21.15.160:6803/4131806045] exists,up 77400671-2855-4f44-911b-98ae802e9162 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 up in weight 1 up_from 26 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6810/678333449,v1:172.21.15.39:6811/678333449] [v2:172.21.15.39:6812/678333449,v1:172.21.15.39:6813/678333449] exists,up 6f906a00-043e-4282-8c8d-47b889db0c65 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 up in weight 1 up_from 32 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6808/1632873600,v1:172.21.15.150:6809/1632873600] [v2:172.21.15.150:6810/1632873600,v1:172.21.15.150:6811/1632873600] exists,up 6a50d102-41d4-41ec-a5b8-1d40420d3cb7 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 up in weight 1 up_from 37 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6808/3500100397,v1:172.21.15.160:6809/3500100397] [v2:172.21.15.160:6810/3500100397,v1:172.21.15.160:6811/3500100397] exists,up 64041b91-4bdc-4e60-b280-2407dc0285f0 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.6 up in weight 1 up_from 43 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6818/1468848111,v1:172.21.15.39:6819/1468848111] [v2:172.21.15.39:6820/1468848111,v1:172.21.15.39:6821/1468848111] exists,up 2e0496c7-05ba-40e2-80fe-18ded790693a 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.7 up in weight 1 up_from 48 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6816/3982567281,v1:172.21.15.150:6817/3982567281] [v2:172.21.15.150:6818/3982567281,v1:172.21.15.150:6819/3982567281] exists,up ec370951-5b96-470f-ad4d-1122615da035 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.8 up in weight 1 up_from 54 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6816/4075583447,v1:172.21.15.160:6817/4075583447] [v2:172.21.15.160:6818/4075583447,v1:172.21.15.160:6819/4075583447] exists,up 289d3eda-511f-4731-9242-02e987aafefd 2024-09-21T01:03:25.017 INFO:teuthology.orchestra.run.smithi039.stdout:osd.9 up in weight 1 up_from 60 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6826/4176461767,v1:172.21.15.39:6827/4176461767] [v2:172.21.15.39:6828/4176461767,v1:172.21.15.39:6829/4176461767] exists,up eb6e99ce-4a27-484a-8850-52b165922de1 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.10 up in weight 1 up_from 65 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6824/2296139693,v1:172.21.15.150:6825/2296139693] [v2:172.21.15.150:6826/2296139693,v1:172.21.15.150:6827/2296139693] exists,up cac8cd3a-cffc-466d-8214-8a3f714f5380 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.11 up in weight 1 up_from 71 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6824/3176075747,v1:172.21.15.160:6825/3176075747] [v2:172.21.15.160:6826/3176075747,v1:172.21.15.160:6827/3176075747] exists,up f0574abe-bba5-4251-bfd6-2610607a2400 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 2.8 [2,11] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 2.f [8,11] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.0 [4,8] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.6 [4,6] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.c [3,9] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.11 [4,6] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.24 [4,9] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.28 [5,7] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.29 [3,9] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.39 [5,8] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.5 [5,10] 2024-09-21T01:03:25.018 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.13 [9,8] 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.19 [9,8] 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.1f [9,8] 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/2061408491 expires 2024-09-22T00:53:07.774291+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/2520758295 expires 2024-09-22T00:51:28.880618+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6800/2575866445 expires 2024-09-22T00:51:58.924468+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6801/368668226 expires 2024-09-22T00:51:28.880618+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3183273760 expires 2024-09-22T00:53:07.774291+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3378583569 expires 2024-09-22T00:51:58.924468+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/412418374 expires 2024-09-22T00:51:28.880618+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6801/2575866445 expires 2024-09-22T00:51:58.924468+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6800/368668226 expires 2024-09-22T00:51:28.880618+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3601289686 expires 2024-09-22T00:51:58.924468+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/712381310 expires 2024-09-22T00:51:58.924468+0000 2024-09-21T01:03:25.019 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3703447041 expires 2024-09-22T00:51:28.880618+0000 2024-09-21T01:03:25.020 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6800/3953053829 expires 2024-09-22T00:53:07.774291+0000 2024-09-21T01:03:25.020 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3894994060 expires 2024-09-22T00:53:07.774291+0000 2024-09-21T01:03:25.020 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6801/3953053829 expires 2024-09-22T00:53:07.774291+0000 2024-09-21T01:03:25.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6b40b65e0 msgr2=0x7ff6b40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:25.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6b40b65e0 0x7ff6b40b8af0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff6d01123b0 tx=0x7ff6b8009260 comp rx=0 tx=0).stop 2024-09-21T01:03:25.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 msgr2=0x7ff6d019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 0x7ff6d019c6c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000ea60 tx=0x7ff6c000ef20 comp rx=0 tx=0).stop 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 shutdown_connections 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6d01100e0 0x7ff6d01a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6b40b65e0 0x7ff6b40b8af0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d0076100 0x7ff6d019cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 --2- 172.21.15.39:0/2817781043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6d00757b0 0x7ff6d019c6c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 >> 172.21.15.39:0/2817781043 conn(0x7ff6d00ff5d0 msgr2=0x7ff6d0109c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 shutdown_connections 2024-09-21T01:03:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:25.018+0000 7ff6d6400640 1 -- 172.21.15.39:0/2817781043 wait complete. 2024-09-21T01:03:25.817 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-21T01:03:25.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:03:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:25 smithi150 ceph-mon[34945]: pgmap v386: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:03:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:25 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2817781043' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T01:03:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:25 smithi160 ceph-mon[34696]: pgmap v386: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:03:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:25 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2817781043' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T01:03:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:25 smithi039 ceph-mon[30415]: pgmap v386: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:03:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:25 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2817781043' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T01:03:26.205 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 -- 172.21.15.39:0/1489571943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d0105130 msgr2=0x7f24d010d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 --2- 172.21.15.39:0/1489571943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d0105130 0x7f24d010d680 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f24b8009900 tx=0x7f24b802a410 comp rx=0 tx=0).stop 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 -- 172.21.15.39:0/1489571943 shutdown_connections 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 --2- 172.21.15.39:0/1489571943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d010dc40 0x7f24d0110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 --2- 172.21.15.39:0/1489571943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d0105130 0x7f24d010d680 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 --2- 172.21.15.39:0/1489571943 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24d01047e0 0x7f24d0104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 -- 172.21.15.39:0/1489571943 >> 172.21.15.39:0/1489571943 conn(0x7f24d00fe300 msgr2=0x7f24d0100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 -- 172.21.15.39:0/1489571943 shutdown_connections 2024-09-21T01:03:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.805+0000 7f24cf200640 1 -- 172.21.15.39:0/1489571943 wait complete. 2024-09-21T01:03:26.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.806+0000 7f24cf200640 1 Processor -- start 2024-09-21T01:03:26.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.806+0000 7f24cf200640 1 -- start start 2024-09-21T01:03:26.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cf200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 0x7f24d0102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cf200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24d0105130 0x7f24d0102a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cf200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d010dc40 0x7f24d0102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cf200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f24d01138f0 con 0x7f24d010dc40 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cf200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24d0113770 con 0x7f24d01047e0 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cf200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f24d0113a70 con 0x7f24d0105130 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cde00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 0x7f24d0102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cde00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 0x7f24d0102520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46422/0 (socket says 172.21.15.39:46422) 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24cde00640 1 -- 172.21.15.39:0/1361417480 learned_addr learned my addr 172.21.15.39:0/1361417480 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24ce800640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d010dc40 0x7f24d0102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:26.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.807+0000 7f24c7e00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24d0105130 0x7f24d0102a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24cde00640 1 -- 172.21.15.39:0/1361417480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24d0105130 msgr2=0x7f24d0102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24cde00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24d0105130 0x7f24d0102a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24cde00640 1 -- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d010dc40 msgr2=0x7f24d0102fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24cde00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d010dc40 0x7f24d0102fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24cde00640 1 -- 172.21.15.39:0/1361417480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d01a85e0 con 0x7f24d01047e0 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24ce800640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d010dc40 0x7f24d0102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:26.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.808+0000 7f24cde00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 0x7f24d0102520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f24c000e9f0 tx=0x7f24c000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.809+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c000cd80 con 0x7f24d01047e0 2024-09-21T01:03:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.809+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c00054d0 con 0x7f24d01047e0 2024-09-21T01:03:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.809+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0010690 con 0x7f24d01047e0 2024-09-21T01:03:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.809+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d01a88d0 con 0x7f24d01047e0 2024-09-21T01:03:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.809+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d01a8d90 con 0x7f24d01047e0 2024-09-21T01:03:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.810+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d0104bf0 con 0x7f24d01047e0 2024-09-21T01:03:26.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.812+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f24c0004e30 con 0x7f24d01047e0 2024-09-21T01:03:26.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.814+0000 7f24c6a00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f24b40b6560 0x7f24b40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:26.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.814+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f24c00e7e50 con 0x7f24d01047e0 2024-09-21T01:03:26.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.814+0000 7f24c7e00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f24b40b6560 0x7f24b40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:26.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.815+0000 7f24c7e00640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f24b40b6560 0x7f24b40b8a70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f24b802aa00 tx=0x7f24b8005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:26.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:26.816+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f24c00ae1e0 con 0x7f24d01047e0 2024-09-21T01:03:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:26 smithi039 ceph-mon[30415]: pgmap v387: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:27.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.052+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24d0103820 con 0x7f24d01047e0 2024-09-21T01:03:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.054+0000 7f24c6a00640 1 -- 172.21.15.39:0/1361417480 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11911 (secure 0 0 0) 0x7f24c00b25c0 con 0x7f24d01047e0 2024-09-21T01:03:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:27.054 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"btime":"2024-09-21T01:01:44:370343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:01:43.492789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:03:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:03:27.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f24b40b6560 msgr2=0x7f24b40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:27.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f24b40b6560 0x7f24b40b8a70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f24b802aa00 tx=0x7f24b8005a20 comp rx=0 tx=0).stop 2024-09-21T01:03:27.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 msgr2=0x7f24d0102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 0x7f24d0102520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f24c000e9f0 tx=0x7f24c000edf0 comp rx=0 tx=0).stop 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 shutdown_connections 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d010dc40 0x7f24d0102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24d0105130 0x7f24d0102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f24b40b6560 0x7f24b40b8a70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 --2- 172.21.15.39:0/1361417480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24d01047e0 0x7f24d0102520 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 >> 172.21.15.39:0/1361417480 conn(0x7f24d00fe300 msgr2=0x7f24d01077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 shutdown_connections 2024-09-21T01:03:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:27.056+0000 7f24cf200640 1 -- 172.21.15.39:0/1361417480 wait complete. 2024-09-21T01:03:27.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:26 smithi150 ceph-mon[34945]: pgmap v387: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:27.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:26 smithi160 ceph-mon[34696]: pgmap v387: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:27.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:03:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:27 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1361417480' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:27 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1361417480' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:27 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1361417480' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 -- 172.21.15.39:0/3826840624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c81049b0 msgr2=0x7f39c8104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3826840624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c81049b0 0x7f39c8104dc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009900 tx=0x7f39b802a410 comp rx=0 tx=0).stop 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 -- 172.21.15.39:0/3826840624 shutdown_connections 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3826840624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c810de10 0x7f39c8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3826840624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 0x7f39c810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3826840624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c81049b0 0x7f39c8104dc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 -- 172.21.15.39:0/3826840624 >> 172.21.15.39:0/3826840624 conn(0x7f39c8079760 msgr2=0x7f39c8079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:28.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 -- 172.21.15.39:0/3826840624 shutdown_connections 2024-09-21T01:03:28.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.937+0000 7f39cfa00640 1 -- 172.21.15.39:0/3826840624 wait complete. 2024-09-21T01:03:28.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.938+0000 7f39cfa00640 1 Processor -- start 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 -- start start 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c81049b0 0x7f39c81026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 0x7f39c8102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 0x7f39c8103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8113a90 con 0x7f39c810de10 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8113910 con 0x7f39c81049b0 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cfa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8113c10 con 0x7f39c8105300 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 0x7f39c8103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 0x7f39c8103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37236/0 (socket says 172.21.15.39:37236) 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cda00640 1 -- 172.21.15.39:0/3000732052 learned_addr learned my addr 172.21.15.39:0/3000732052 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 0x7f39c8102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:28.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.939+0000 7f39cd000640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c81049b0 0x7f39c81026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:28.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39bfe00640 1 -- 172.21.15.39:0/3000732052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c81049b0 msgr2=0x7f39c81026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:28.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39bfe00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c81049b0 0x7f39c81026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:28.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39bfe00640 1 -- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 msgr2=0x7f39c8103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:28.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39bfe00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 0x7f39c8103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39bfe00640 1 -- 172.21.15.39:0/3000732052 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81a8760 con 0x7f39c8105300 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39cda00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 0x7f39c8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39bfe00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 0x7f39c8102c30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39c806b950 tx=0x7f39b000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b000cc40 con 0x7f39c8105300 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39b000cde0 con 0x7f39c8105300 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.940+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0004a00 con 0x7f39c8105300 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.941+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c81a8a50 con 0x7f39c8105300 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.941+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c81a8f40 con 0x7f39c8105300 2024-09-21T01:03:28.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.942+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c806c2c0 con 0x7f39c8105300 2024-09-21T01:03:28.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.944+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f39b0004ba0 con 0x7f39c8105300 2024-09-21T01:03:28.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.946+0000 7f39c6000640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39ac0b6560 0x7f39ac0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:28.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.946+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f39b00e7b60 con 0x7f39c8105300 2024-09-21T01:03:28.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.946+0000 7f39cd000640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39ac0b6560 0x7f39ac0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:28.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.947+0000 7f39cd000640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39ac0b6560 0x7f39ac0b8a70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f39b8000c00 tx=0x7f39b8031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:28.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:28.947+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f39b00adef0 con 0x7f39c8105300 2024-09-21T01:03:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:28 smithi150 ceph-mon[34945]: pgmap v388: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:29.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:28 smithi160 ceph-mon[34696]: pgmap v388: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:29.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:28 smithi039 ceph-mon[30415]: pgmap v388: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.191+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39c8104dc0 con 0x7f39c8105300 2024-09-21T01:03:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.192+0000 7f39c6000640 1 -- 172.21.15.39:0/3000732052 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11911 (secure 0 0 0) 0x7f39c8104dc0 con 0x7f39c8105300 2024-09-21T01:03:29.193 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:29.193 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"btime":"2024-09-21T01:01:44:370343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:01:43.492789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:03:29.193 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.194+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39ac0b6560 msgr2=0x7f39ac0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.194+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39ac0b6560 0x7f39ac0b8a70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f39b8000c00 tx=0x7f39b8031400 comp rx=0 tx=0).stop 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.194+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 msgr2=0x7f39c8102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 0x7f39c8102c30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39c806b950 tx=0x7f39b000ee50 comp rx=0 tx=0).stop 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 shutdown_connections 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c810de10 0x7f39c8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39c8105300 0x7f39c8102c30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:29.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39ac0b6560 0x7f39ac0b8a70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:29.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 --2- 172.21.15.39:0/3000732052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c81049b0 0x7f39c81026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:29.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 >> 172.21.15.39:0/3000732052 conn(0x7f39c8079760 msgr2=0x7f39c8107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:29.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 shutdown_connections 2024-09-21T01:03:29.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:29.195+0000 7f39cfa00640 1 -- 172.21.15.39:0/3000732052 wait complete. 2024-09-21T01:03:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:29 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3000732052' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:30.026 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:03:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:29 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3000732052' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:29 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3000732052' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:30.405 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:30.804 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:30 smithi039 ceph-mon[30415]: pgmap v389: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:30 smithi150 ceph-mon[34945]: pgmap v389: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 -- 172.21.15.39:0/571356291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc410b6c0 msgr2=0x7f1cc410bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 --2- 172.21.15.39:0/571356291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc410b6c0 0x7f1cc410bad0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009860 tx=0x7f1cac02a340 comp rx=0 tx=0).stop 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 -- 172.21.15.39:0/571356291 shutdown_connections 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 --2- 172.21.15.39:0/571356291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cc410c010 0x7f1cc4112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 --2- 172.21.15.39:0/571356291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc410b6c0 0x7f1cc410bad0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 --2- 172.21.15.39:0/571356291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc4105d80 0x7f1cc4106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 -- 172.21.15.39:0/571356291 >> 172.21.15.39:0/571356291 conn(0x7f1cc4101810 msgr2=0x7f1cc4103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 -- 172.21.15.39:0/571356291 shutdown_connections 2024-09-21T01:03:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.086+0000 7f1cca600640 1 -- 172.21.15.39:0/571356291 wait complete. 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.087+0000 7f1cca600640 1 Processor -- start 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.087+0000 7f1cca600640 1 -- start start 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cca600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc4105d80 0x7f1cc419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cca600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 0x7f1cc419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cca600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cc410c010 0x7f1cc41a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cca600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc4115cd0 con 0x7f1cc410b6c0 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cca600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc4115b50 con 0x7f1cc410c010 2024-09-21T01:03:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cca600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc4115e50 con 0x7f1cc4105d80 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cbbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 0x7f1cc419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cbbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 0x7f1cc419cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37256/0 (socket says 172.21.15.39:37256) 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cc3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cc410c010 0x7f1cc41a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cbbe00640 1 -- 172.21.15.39:0/1751449516 learned_addr learned my addr 172.21.15.39:0/1751449516 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.088+0000 7f1cc3400640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc4105d80 0x7f1cc419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cbbe00640 1 -- 172.21.15.39:0/1751449516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc4105d80 msgr2=0x7f1cc419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cbbe00640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc4105d80 0x7f1cc419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cbbe00640 1 -- 172.21.15.39:0/1751449516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cc410c010 msgr2=0x7f1cc41a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cbbe00640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cc410c010 0x7f1cc41a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cbbe00640 1 -- 172.21.15.39:0/1751449516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc41a2630 con 0x7f1cc410b6c0 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cc3400640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc4105d80 0x7f1cc419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cbbe00640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 0x7f1cc419cc60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1cac035040 tx=0x7f1cac032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac037040 con 0x7f1cc410b6c0 2024-09-21T01:03:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc41a28c0 con 0x7f1cc410b6c0 2024-09-21T01:03:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cac030c10 con 0x7f1cc410b6c0 2024-09-21T01:03:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc41a9080 con 0x7f1cc410b6c0 2024-09-21T01:03:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.089+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac0035f0 con 0x7f1cc410b6c0 2024-09-21T01:03:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.091+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc4107000 con 0x7f1cc410b6c0 2024-09-21T01:03:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.095+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f1cac043020 con 0x7f1cc410b6c0 2024-09-21T01:03:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.096+0000 7f1cc0c00640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca80b6560 0x7f1ca80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.096+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1cac107120 con 0x7f1cc410b6c0 2024-09-21T01:03:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.097+0000 7f1cc3400640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca80b6560 0x7f1ca80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.097+0000 7f1cc3400640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca80b6560 0x7f1ca80b8a70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40096c0 tx=0x7f1cb4009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:31.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.098+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1cac0cd4b0 con 0x7f1cc410b6c0 2024-09-21T01:03:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:30 smithi160 ceph-mon[34696]: pgmap v389: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:31.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.297+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cc419d7f0 con 0x7f1cc410b6c0 2024-09-21T01:03:31.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.299+0000 7f1cc0c00640 1 -- 172.21.15.39:0/1751449516 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25331 (secure 0 0 0) 0x7f1cac030dc0 con 0x7f1cc410b6c0 2024-09-21T01:03:31.300 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:31.300 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":90,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:02:10.480352+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca80b6560 msgr2=0x7f1ca80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca80b6560 0x7f1ca80b8a70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40096c0 tx=0x7f1cb4009260 comp rx=0 tx=0).stop 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 msgr2=0x7f1cc419cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 0x7f1cc419cc60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1cac035040 tx=0x7f1cac032d50 comp rx=0 tx=0).stop 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 shutdown_connections 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1ca80b6560 0x7f1ca80b8a70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cc410c010 0x7f1cc41a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc410b6c0 0x7f1cc419cc60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 --2- 172.21.15.39:0/1751449516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cc4105d80 0x7f1cc419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:31.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.303+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 >> 172.21.15.39:0/1751449516 conn(0x7f1cc4101810 msgr2=0x7f1cc4103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:31.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.304+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 shutdown_connections 2024-09-21T01:03:31.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:31.304+0000 7f1cca600640 1 -- 172.21.15.39:0/1751449516 wait complete. 2024-09-21T01:03:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:31 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1751449516' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:03:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:31 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1751449516' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:03:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:31 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1751449516' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:03:32.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T01:00:43.758917+0000', 'modified': '2024-09-21T01:01:43.492789+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24764, 'mds_1': 24760, 'mds_2': 24742}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24715': {'gid': 24715, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6835/2680876155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6834', 'nonce': 2680876155}, {'type': 'v1', 'addr': '172.21.15.160:6835', 'nonce': 2680876155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24725': {'gid': 24725, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/555411539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 555411539}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 555411539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24727': {'gid': 24727, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6837/1965195431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6836', 'nonce': 1965195431}, {'type': 'v1', 'addr': '172.21.15.160:6837', 'nonce': 1965195431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24742': {'gid': 24742, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.160:6839/2403371273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6838', 'nonce': 2403371273}, {'type': 'v1', 'addr': '172.21.15.160:6839', 'nonce': 2403371273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24760': {'gid': 24760, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.39:6841/3868264331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6840', 'nonce': 3868264331}, {'type': 'v1', 'addr': '172.21.15.39:6841', 'nonce': 3868264331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24764': {'gid': 24764, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.150:6839/103174700', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 103174700}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 103174700}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24764, 'qdb_cluster': [24764, 24760, 24742]} 2024-09-21T01:03:32.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-21T01:03:32.246 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T01:03:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:33.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:32 smithi150 ceph-mon[34945]: pgmap v390: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:33.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:32 smithi160 ceph-mon[34696]: pgmap v390: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:33.116 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:32 smithi039 ceph-mon[30415]: pgmap v390: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 -- 172.21.15.39:0/4085553754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8109e10 msgr2=0x7fe6c8110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/4085553754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8109e10 0x7fe6c8110640 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009ca0 tx=0x7fe6c002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 -- 172.21.15.39:0/4085553754 shutdown_connections 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/4085553754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8109e10 0x7fe6c8110640 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/4085553754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6c81094c0 0x7fe6c81098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/4085553754 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6c8103b80 0x7fe6c8103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 -- 172.21.15.39:0/4085553754 >> 172.21.15.39:0/4085553754 conn(0x7fe6c80ff5d0 msgr2=0x7fe6c8101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 -- 172.21.15.39:0/4085553754 shutdown_connections 2024-09-21T01:03:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.295+0000 7fe6cfc00640 1 -- 172.21.15.39:0/4085553754 wait complete. 2024-09-21T01:03:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 Processor -- start 2024-09-21T01:03:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 -- start start 2024-09-21T01:03:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 0x7fe6c8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6c81094c0 0x7fe6c8075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6c8109e10 0x7fe6c8076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c8113ad0 con 0x7fe6c8103b80 2024-09-21T01:03:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c8113950 con 0x7fe6c81094c0 2024-09-21T01:03:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.297+0000 7fe6cfc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c8113c50 con 0x7fe6c8109e10 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 0x7fe6c8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 0x7fe6c8075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51244/0 (socket says 172.21.15.39:51244) 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 -- 172.21.15.39:0/2336733966 learned_addr learned my addr 172.21.15.39:0/2336733966 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6c8109e10 0x7fe6c8076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6c81094c0 0x7fe6c8075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 -- 172.21.15.39:0/2336733966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6c8109e10 msgr2=0x7fe6c8076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6c8109e10 0x7fe6c8076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 -- 172.21.15.39:0/2336733966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6c81094c0 msgr2=0x7fe6c8075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6c81094c0 0x7fe6c8075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 -- 172.21.15.39:0/2336733966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c81a8700 con 0x7fe6c8103b80 2024-09-21T01:03:33.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.298+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 0x7fe6c8075710 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e9f0 tx=0x7fe6bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:33.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.299+0000 7fe6c5600640 1 -- 172.21.15.39:0/2336733966 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc00cd80 con 0x7fe6c8103b80 2024-09-21T01:03:33.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.299+0000 7fe6c5600640 1 -- 172.21.15.39:0/2336733966 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6bc0054d0 con 0x7fe6c8103b80 2024-09-21T01:03:33.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.299+0000 7fe6c5600640 1 -- 172.21.15.39:0/2336733966 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc010690 con 0x7fe6c8103b80 2024-09-21T01:03:33.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.299+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c81a89f0 con 0x7fe6c8103b80 2024-09-21T01:03:33.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.299+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c81a8ee0 con 0x7fe6c8103b80 2024-09-21T01:03:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.302+0000 7fe6d6840740 1 Processor -- start 2024-09-21T01:03:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.302+0000 7fe6d6840740 1 -- start start 2024-09-21T01:03:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6c5600640 1 -- 172.21.15.39:0/2336733966 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7fe6bc004e90 con 0x7fe6c8103b80 2024-09-21T01:03:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2d6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2d69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2d6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6d6840740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5622df24dd50 con 0x5622def0f280 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6d6840740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5622df23d180 con 0x5622def070a0 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6d6840740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5622df240f90 con 0x5622def2ff60 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2d6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2d69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2d6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51248/0 (socket says 172.21.15.39:51248) 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6c7e00640 1 -- 172.21.15.39:0/2836325802 learned_addr learned my addr 172.21.15.39:0/2836325802 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.303+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2d6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6cd200640 1 -- 172.21.15.39:0/2836325802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 msgr2=0x5622df2d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2d6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6cd200640 1 -- 172.21.15.39:0/2836325802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 msgr2=0x5622df2d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2d6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6cd200640 1 -- 172.21.15.39:0/2836325802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622df207a70 con 0x5622def070a0 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2d6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2d69d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0e5180 tx=0x7fe6bc0ebc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0b012872c633273 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6ba000640 1 -- 172.21.15.39:0/2836325802 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0ebe60 con 0x5622def070a0 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6ba000640 1 -- 172.21.15.39:0/2836325802 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6bc0f9070 con 0x5622def070a0 2024-09-21T01:03:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.304+0000 7fe6ba000640 1 -- 172.21.15.39:0/2836325802 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0f4dd0 con 0x5622def070a0 2024-09-21T01:03:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.305+0000 7fe6c5600640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe6b00b65e0 0x7fe6b00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.305+0000 7fe6d6840740 1 -- 172.21.15.39:0/2836325802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 msgr2=0x5622df2d69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.305+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2d69d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0e5180 tx=0x7fe6bc0ebc10 comp rx=0 tx=0).stop 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.305+0000 7fe6c5600640 1 -- 172.21.15.39:0/2336733966 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe6bc0e7be0 con 0x7fe6c8103b80 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.305+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe6b00b65e0 0x7fe6b00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 -- 172.21.15.39:0/2836325802 shutdown_connections 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2d6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2d69d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2836325802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2d6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 -- 172.21.15.39:0/2836325802 >> 172.21.15.39:0/2836325802 conn(0x5622def24350 msgr2=0x5622df2d4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 -- 172.21.15.39:0/2836325802 shutdown_connections 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 -- 172.21.15.39:0/2836325802 wait complete. 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe6b00b65e0 0x7fe6b00b8af0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0006010 tx=0x7fe6c0009920 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.306+0000 7fe6d6840740 1 -- start start 2024-09-21T01:03:33.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 Processor -- start 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2e2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2edf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5622df09e5f0 con 0x5622def0f280 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5622df1d8460 con 0x5622def070a0 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5622df2e4c10 con 0x5622def2ff60 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.307+0000 7fe6d6840740 10 client.0 _ll_register_callbacks cb 0x5622df1e8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2e2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2e2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51264/0 (socket says 172.21.15.39:51264) 2024-09-21T01:03:33.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6c7e00640 1 -- 172.21.15.39:0/817355607 learned_addr learned my addr 172.21.15.39:0/817355607 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2e1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:40206/0 (socket says 172.21.15.39:40206) 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 -- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 msgr2=0x5622df2edf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2edf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 -- 172.21.15.39:0/817355607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 msgr2=0x5622df2e2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2e2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 -- 172.21.15.39:0/817355607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622def9b540 con 0x5622def070a0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2e2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6cd200640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2e1fe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e9c0 tx=0x7fe6bc00cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c34ad4e13e9404d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.308+0000 7fe6aea00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6aea00640 1 -- 172.21.15.39:0/817355607 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0a9ec0 con 0x5622def070a0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6aea00640 1 -- 172.21.15.39:0/817355607 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe6bc0b6e00 con 0x5622def070a0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5622df1daa70 con 0x5622def070a0 2024-09-21T01:03:33.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6aea00640 1 -- 172.21.15.39:0/817355607 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe6bc005180 con 0x5622def070a0 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6d6840740 10 client.25322 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622df1f56f0 con 0x5622def070a0 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6aea00640 1 -- 172.21.15.39:0/817355607 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5622df1daa70 con 0x5622def070a0 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.309+0000 7fe6aea00640 1 -- 172.21.15.39:0/817355607 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe6bc0b0c60 con 0x5622def070a0 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6d6840740 10 client.25322 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6d6840740 10 client.25322 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6d6840740 20 client.25322 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6d6840740 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deeee8e0 0x5622df2f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6d6840740 4 client.25322 mds_command: new command op to 24742 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:03:33.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 --> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622df24fca0 con 0x5622deeee8e0 2024-09-21T01:03:33.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.310+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deeee8e0 0x5622df2f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.311+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deeee8e0 0x5622df2f1db0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70cc2afe95066503 server_cookie=ddea6c411b2c438e in_seq=0 out_seq=0 2024-09-21T01:03:33.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.311+0000 7fe6aea00640 10 client.25322 ms_handle_connect on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6aea00640 1 -- 172.21.15.39:0/817355607 <== mds.2 v2:172.21.15.160:6838/2403371273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5622df24fca0 con 0x5622deeee8e0 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6aea00640 10 client.25322 handle_command_reply: tid=0 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 client.25322 shutdown 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 msgr2=0x5622df2e1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2e1fe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e9c0 tx=0x7fe6bc00cf20 comp rx=0 tx=0).stop 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 shutdown_connections 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deeee8e0 0x5622df2f1db0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2edf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2e2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 --2- 172.21.15.39:0/817355607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def070a0 0x5622df2e1fe0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6aea00640 0 client.25322 ms_handle_reset on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 >> 172.21.15.39:0/817355607 conn(0x5622def24350 msgr2=0x5622df2d9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 shutdown_connections 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 1 -- 172.21.15.39:0/817355607 wait complete. 2024-09-21T01:03:33.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.312+0000 7fe6d6840740 20 client.25322 trim_cache size 0 max 16384 2024-09-21T01:03:33.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 Processor -- start 2024-09-21T01:03:33.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 -- start start 2024-09-21T01:03:33.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def070a0 0x5622df2f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2eded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def2ff60 0x5622df2d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5622df2f2300 con 0x5622def0f280 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5622df2f29d0 con 0x5622def2ff60 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6d6840740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5622df2f30a0 con 0x5622def070a0 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2eded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def070a0 0x5622df2f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2eded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51272/0 (socket says 172.21.15.39:51272) 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.332+0000 7fe6c7e00640 1 -- 172.21.15.39:0/2714293304 learned_addr learned my addr 172.21.15.39:0/2714293304 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def2ff60 0x5622df2d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6c7e00640 1 -- 172.21.15.39:0/2714293304 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def070a0 msgr2=0x5622df2f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def070a0 0x5622df2f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6c7e00640 1 -- 172.21.15.39:0/2714293304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def2ff60 msgr2=0x5622df2d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def2ff60 0x5622df2d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6c7e00640 1 -- 172.21.15.39:0/2714293304 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622df2d92f0 con 0x5622def0f280 2024-09-21T01:03:33.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6cd200640 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def070a0 0x5622df2f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:33.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.333+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2eded0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5622df2e2e50 tx=0x7fe6b4009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728db6f32196f8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:33.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6b9600640 1 -- 172.21.15.39:0/2714293304 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4020040 con 0x5622def0f280 2024-09-21T01:03:33.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6b9600640 1 -- 172.21.15.39:0/2714293304 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6b400fd40 con 0x5622def0f280 2024-09-21T01:03:33.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6b9600640 1 -- 172.21.15.39:0/2714293304 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4003b20 con 0x5622def0f280 2024-09-21T01:03:33.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 -- 172.21.15.39:0/2714293304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 msgr2=0x5622df2eded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2eded0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5622df2e2e50 tx=0x7fe6b4009f60 comp rx=0 tx=0).stop 2024-09-21T01:03:33.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 -- 172.21.15.39:0/2714293304 shutdown_connections 2024-09-21T01:03:33.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def2ff60 0x5622df2d3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def0f280 0x5622df2eded0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 --2- 172.21.15.39:0/2714293304 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def070a0 0x5622df2f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.334+0000 7fe6d6840740 1 -- 172.21.15.39:0/2714293304 >> 172.21.15.39:0/2714293304 conn(0x5622def24350 msgr2=0x5622df2e1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.335+0000 7fe6d6840740 1 -- 172.21.15.39:0/2714293304 shutdown_connections 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.335+0000 7fe6d6840740 1 -- 172.21.15.39:0/2714293304 wait complete. 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.335+0000 7fe6d6840740 1 -- start start 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.335+0000 7fe6d6840740 1 Processor -- start 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 0x5622df2fc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def0f280 0x5622df2f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2f6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5622df2f3e40 con 0x5622def070a0 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5622df2f3770 con 0x5622def0f280 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5622df2f4510 con 0x5622def2ff60 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6d6840740 10 client.0 _ll_register_callbacks cb 0x7fe6bc0eb9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 0x5622df2fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 0x5622df2fc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51276/0 (socket says 172.21.15.39:51276) 2024-09-21T01:03:33.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6cd200640 1 -- 172.21.15.39:0/792547184 learned_addr learned my addr 172.21.15.39:0/792547184 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:33.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def0f280 0x5622df2f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.336+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cd200640 1 -- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 msgr2=0x5622df2f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cd200640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cd200640 1 -- 172.21.15.39:0/792547184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def0f280 msgr2=0x5622df2f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cd200640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def0f280 0x5622df2f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cd200640 1 -- 172.21.15.39:0/792547184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622df2f67d0 con 0x5622def070a0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cdc00640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6cd200640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 0x5622df2fc850 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0b0fd0 tx=0x7fe6bc0e7e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8017b6672d1678 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6aea00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6aea00640 1 -- 172.21.15.39:0/792547184 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0ebe60 con 0x5622def070a0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6aea00640 1 -- 172.21.15.39:0/792547184 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe6bc0b7000 con 0x5622def070a0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.337+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b400fd40 con 0x5622def070a0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6aea00640 1 -- 172.21.15.39:0/792547184 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe6bc0acfe0 con 0x5622def070a0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 10 client.15318 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622df2fc1d0 con 0x5622def070a0 2024-09-21T01:03:33.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6aea00640 1 -- 172.21.15.39:0/792547184 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6b400fd40 con 0x5622def070a0 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6aea00640 1 -- 172.21.15.39:0/792547184 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe6bc0aeba0 con 0x5622def070a0 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 10 client.15318 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 10 client.15318 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 20 client.15318 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deec3c90 0x5622df3628d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 4 client.15318 mds_command: new command op to 24742 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.338+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 --> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] -- command(tid 0: {"prefix": "status"}) -- 0x5622df24b750 con 0x5622deec3c90 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.339+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deec3c90 0x5622df3628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.339+0000 7fe6c7e00640 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deec3c90 0x5622df3628d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e95a213b9312cf32 server_cookie=5a88e7c7ba2fbb47 in_seq=0 out_seq=0 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.339+0000 7fe6aea00640 10 client.15318 ms_handle_connect on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.339+0000 7fe6aea00640 1 -- 172.21.15.39:0/792547184 <== mds.2 v2:172.21.15.160:6838/2403371273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5622df24b750 con 0x5622deec3c90 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.339+0000 7fe6aea00640 10 client.15318 handle_command_reply: tid=0 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.339+0000 7fe6d6840740 1 client.15318 shutdown 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 msgr2=0x5622df2fc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 0x5622df2fc850 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0b0fd0 tx=0x7fe6bc0e7e60 comp rx=0 tx=0).stop 2024-09-21T01:03:33.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 shutdown_connections 2024-09-21T01:03:33.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x5622deec3c90 0x5622df3628d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5622def2ff60 0x5622df2f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622def0f280 0x5622df2f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 --2- 172.21.15.39:0/792547184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5622def070a0 0x5622df2fc850 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 >> 172.21.15.39:0/792547184 conn(0x5622def24350 msgr2=0x5622df2d94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6aea00640 0 client.15318 ms_handle_reset on v2:172.21.15.160:6838/2403371273 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 shutdown_connections 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 1 -- 172.21.15.39:0/792547184 wait complete. 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.340+0000 7fe6d6840740 20 client.15318 trim_cache size 0 max 16384 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 2, 2024-09-21T01:03:33.344 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24742, 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 115.00007252899999, 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 193.99534300900001 2024-09-21T01:03:33.345 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe6b00b65e0 msgr2=0x7fe6b00b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe6b00b65e0 0x7fe6b00b8af0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0006010 tx=0x7fe6c0009920 comp rx=0 tx=0).stop 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 msgr2=0x7fe6c8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 0x7fe6c8075710 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e9f0 tx=0x7fe6bc00edf0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 shutdown_connections 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6c8109e10 0x7fe6c8076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe6b00b65e0 0x7fe6b00b8af0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6c81094c0 0x7fe6c8075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 --2- 172.21.15.39:0/2336733966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6c8103b80 0x7fe6c8075710 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 >> 172.21.15.39:0/2336733966 conn(0x7fe6c80ff5d0 msgr2=0x7fe6c81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:33.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 shutdown_connections 2024-09-21T01:03:33.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:33.341+0000 7fe6cfc00640 1 -- 172.21.15.39:0/2336733966 wait complete. 2024-09-21T01:03:34.352 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.a status 2024-09-21T01:03:34.730 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:35.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:34 smithi039 ceph-mon[30415]: pgmap v391: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:35.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:34 smithi150 ceph-mon[34945]: pgmap v391: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:35.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:34 smithi160 ceph-mon[34696]: pgmap v391: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:35.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 -- 172.21.15.39:0/3371773706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 msgr2=0x7f6e9c077d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/3371773706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c077d50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009900 tx=0x7f6e9002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 -- 172.21.15.39:0/3371773706 shutdown_connections 2024-09-21T01:03:35.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/3371773706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e9c114570 0x7f6e9c116990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/3371773706 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c078290 0x7f6e9c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/3371773706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c077d50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 -- 172.21.15.39:0/3371773706 >> 172.21.15.39:0/3371773706 conn(0x7f6e9c101680 msgr2=0x7f6e9c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:35.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 -- 172.21.15.39:0/3371773706 shutdown_connections 2024-09-21T01:03:35.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.374+0000 7f6ea2e00640 1 -- 172.21.15.39:0/3371773706 wait complete. 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 Processor -- start 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 -- start start 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c078290 0x7f6e9c1a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e9c114570 0x7f6e9c1a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c119ed0 con 0x7f6e9c077940 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c119d50 con 0x7f6e9c114570 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.375+0000 7f6ea2e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c11a050 con 0x7f6e9c078290 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c1a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51288/0 (socket says 172.21.15.39:51288) 2024-09-21T01:03:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 -- 172.21.15.39:0/2103805110 learned_addr learned my addr 172.21.15.39:0/2103805110 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1000640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c078290 0x7f6e9c1a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea2400640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e9c114570 0x7f6e9c1a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 -- 172.21.15.39:0/2103805110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c078290 msgr2=0x7f6e9c1a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c078290 0x7f6e9c1a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 -- 172.21.15.39:0/2103805110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e9c114570 msgr2=0x7f6e9c1a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e9c114570 0x7f6e9c1a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.376+0000 7f6ea1a00640 1 -- 172.21.15.39:0/2103805110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9c1a6810 con 0x7f6e9c077940 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.377+0000 7f6ea1a00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c1a08a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e90005940 tx=0x7f6e90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.377+0000 7f6e9a000640 1 -- 172.21.15.39:0/2103805110 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9003f030 con 0x7f6e9c077940 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.377+0000 7f6e9a000640 1 -- 172.21.15.39:0/2103805110 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e90003aa0 con 0x7f6e9c077940 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.377+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9c1a6aa0 con 0x7f6e9c077940 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.377+0000 7f6e9a000640 1 -- 172.21.15.39:0/2103805110 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90035070 con 0x7f6e9c077940 2024-09-21T01:03:35.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.377+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e9c1ad260 con 0x7f6e9c077940 2024-09-21T01:03:35.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.381+0000 7f6eab02e740 1 Processor -- start 2024-09-21T01:03:35.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 -- start start 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde82670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde82bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde830f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5580fddf9d50 con 0x5580fdab30a0 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5580fdde9180 con 0x5580fdabb280 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6eab02e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5580fddecf90 con 0x5580fdadbf60 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde82670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde82670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:47472/0 (socket says 172.21.15.39:47472) 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.382+0000 7f6ea2400640 1 -- 172.21.15.39:0/1970101061 learned_addr learned my addr 172.21.15.39:0/1970101061 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6e9a000640 1 -- 172.21.15.39:0/2103805110 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f6e90042030 con 0x7f6e9c077940 2024-09-21T01:03:35.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea1000640 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde830f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea2400640 1 -- 172.21.15.39:0/1970101061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 msgr2=0x5580fde830f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde830f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea2400640 1 -- 172.21.15.39:0/1970101061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 msgr2=0x5580fde82bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde82bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea2400640 1 -- 172.21.15.39:0/1970101061 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580fddb3a70 con 0x5580fdadbf60 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.383+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde82670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009fc0 tx=0x7f6e94010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39cbe31d8a287d9c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:35.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.384+0000 7f6e8ea00640 1 -- 172.21.15.39:0/1970101061 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94003b70 con 0x5580fdadbf60 2024-09-21T01:03:35.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.384+0000 7f6e8ea00640 1 -- 172.21.15.39:0/1970101061 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e94003d10 con 0x5580fdadbf60 2024-09-21T01:03:35.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.384+0000 7f6e8ea00640 1 -- 172.21.15.39:0/1970101061 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94011a40 con 0x5580fdadbf60 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.384+0000 7f6e9a000640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6e840b6560 0x7f6e840b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.384+0000 7f6eab02e740 1 -- 172.21.15.39:0/1970101061 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 msgr2=0x5580fde82670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.384+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde82670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009fc0 tx=0x7f6e94010860 comp rx=0 tx=0).stop 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6e9a000640 1 -- 172.21.15.39:0/2103805110 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6e90107d80 con 0x7f6e9c077940 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 -- 172.21.15.39:0/1970101061 shutdown_connections 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6ea1a00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6e840b6560 0x7f6e840b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde830f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde82bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1970101061 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde82670 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 -- 172.21.15.39:0/1970101061 >> 172.21.15.39:0/1970101061 conn(0x5580fdad0350 msgr2=0x5580fde80560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:35.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 -- 172.21.15.39:0/1970101061 shutdown_connections 2024-09-21T01:03:35.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6eab02e740 1 -- 172.21.15.39:0/1970101061 wait complete. 2024-09-21T01:03:35.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.385+0000 7f6ea1a00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6e840b6560 0x7f6e840b8a70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6e900cd9a0 tx=0x7f6e90031950 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:35.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 -- start start 2024-09-21T01:03:35.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 Processor -- start 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde8e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde8e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde9a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5580fdc4a5f0 con 0x5580fdab30a0 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5580fdd84460 con 0x5580fdabb280 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5580fde90e50 con 0x5580fdadbf60 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.386+0000 7f6eab02e740 10 client.0 _ll_register_callbacks cb 0x5580fdd94210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde8e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde8e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51310/0 (socket says 172.21.15.39:51310) 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea1000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde8e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea2400640 1 -- 172.21.15.39:0/3622516828 learned_addr learned my addr 172.21.15.39:0/3622516828 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea2400640 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde9a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea1000640 1 -- 172.21.15.39:0/3622516828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 msgr2=0x5580fde9a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea1000640 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde9a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea1000640 1 -- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 msgr2=0x5580fde8e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea1000640 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde8e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.387+0000 7f6ea1000640 1 -- 172.21.15.39:0/3622516828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580fdb47540 con 0x5580fdabb280 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6ea1000640 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde8e780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c079b00 tx=0x7f6e8800be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73cb97d7278ea82b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:35.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6e83400640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6e83400640 1 -- 172.21.15.39:0/3622516828 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88005020 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5580fdd86a70 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6e83400640 1 -- 172.21.15.39:0/3622516828 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6e880051c0 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6e83400640 1 -- 172.21.15.39:0/3622516828 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e88023050 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6eab02e740 10 client.25340 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.388+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5580fdda16f0 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6e83400640 1 -- 172.21.15.39:0/3622516828 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5580fdd86a70 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6e83400640 1 -- 172.21.15.39:0/3622516828 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6e88019030 con 0x5580fdabb280 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6eab02e740 10 client.25340 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6eab02e740 10 client.25340 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6eab02e740 20 client.25340 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6eab02e740 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda9a8e0 0x5580fdea0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6eab02e740 4 client.25340 mds_command: new command op to 24760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:03:35.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 --> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580fddfbca0 con 0x5580fda9a8e0 2024-09-21T01:03:35.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.389+0000 7f6ea2400640 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda9a8e0 0x5580fdea0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.390+0000 7f6ea2400640 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda9a8e0 0x5580fdea0740 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b45557a3cc97712 server_cookie=9f02524c26cb5e43 in_seq=0 out_seq=0 2024-09-21T01:03:35.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.390+0000 7f6e83400640 10 client.25340 ms_handle_connect on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:35.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6e83400640 1 -- 172.21.15.39:0/3622516828 <== mds.1 v2:172.21.15.39:6840/3868264331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5580fddfbca0 con 0x5580fda9a8e0 2024-09-21T01:03:35.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6e83400640 10 client.25340 handle_command_reply: tid=0 2024-09-21T01:03:35.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 client.25340 shutdown 2024-09-21T01:03:35.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 msgr2=0x5580fde8e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde8e780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c079b00 tx=0x7f6e8800be80 comp rx=0 tx=0).stop 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 shutdown_connections 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda9a8e0 0x5580fdea0740 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdadbf60 0x5580fde9a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdabb280 0x5580fde8e780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6e83400640 0 client.25340 ms_handle_reset on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 --2- 172.21.15.39:0/3622516828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde8e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 >> 172.21.15.39:0/3622516828 conn(0x5580fdad0350 msgr2=0x5580fde85800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 shutdown_connections 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 1 -- 172.21.15.39:0/3622516828 wait complete. 2024-09-21T01:03:35.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.391+0000 7f6eab02e740 20 client.25340 trim_cache size 0 max 16384 2024-09-21T01:03:35.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.410+0000 7f6eab02e740 1 Processor -- start 2024-09-21T01:03:35.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 -- start start 2024-09-21T01:03:35.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde9e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fde891f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdadbf60 0x5580fde89730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5580fde9ea80 con 0x5580fdab30a0 2024-09-21T01:03:35.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5580fde9f150 con 0x5580fdadbf60 2024-09-21T01:03:35.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6eab02e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5580fde9f820 con 0x5580fdabb280 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde9e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde9e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51324/0 (socket says 172.21.15.39:51324) 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6ea2400640 1 -- 172.21.15.39:0/1382528354 learned_addr learned my addr 172.21.15.39:0/1382528354 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.411+0000 7f6ea1000640 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fde891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdadbf60 0x5580fde89730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 -- 172.21.15.39:0/1382528354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 msgr2=0x5580fde891f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fde891f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 -- 172.21.15.39:0/1382528354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdadbf60 msgr2=0x5580fde89730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdadbf60 0x5580fde89730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 -- 172.21.15.39:0/1382528354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580fde84960 con 0x5580fdab30a0 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde9e190 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009f60 tx=0x7f6e940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d1725f9413ab19 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6e8e000640 1 -- 172.21.15.39:0/1382528354 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94037070 con 0x5580fdab30a0 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6e8e000640 1 -- 172.21.15.39:0/1382528354 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e9402c450 con 0x5580fdab30a0 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.412+0000 7f6e8e000640 1 -- 172.21.15.39:0/1382528354 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94032690 con 0x5580fdab30a0 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 -- 172.21.15.39:0/1382528354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 msgr2=0x5580fde9e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde9e190 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009f60 tx=0x7f6e940115a0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 -- 172.21.15.39:0/1382528354 shutdown_connections 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdadbf60 0x5580fde89730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fde891f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1382528354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdab30a0 0x5580fde9e190 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 -- 172.21.15.39:0/1382528354 >> 172.21.15.39:0/1382528354 conn(0x5580fdad0350 msgr2=0x5580fde9a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 -- 172.21.15.39:0/1382528354 shutdown_connections 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.413+0000 7f6eab02e740 1 -- 172.21.15.39:0/1382528354 wait complete. 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.414+0000 7f6eab02e740 1 -- start start 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.414+0000 7f6eab02e740 1 Processor -- start 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdab30a0 0x5580fdea8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fdea23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 0x5580fdea28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5580fdea05c0 con 0x5580fdadbf60 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5580fde9fef0 con 0x5580fdab30a0 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5580fdea0c90 con 0x5580fdabb280 2024-09-21T01:03:35.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6eab02e740 10 client.0 _ll_register_callbacks cb 0x7f6e88004370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdab30a0 0x5580fdea8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 0x5580fdea28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea1000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fdea23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 0x5580fdea28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51332/0 (socket says 172.21.15.39:51332) 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea2400640 1 -- 172.21.15.39:0/1791660699 learned_addr learned my addr 172.21.15.39:0/1791660699 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea1000640 1 -- 172.21.15.39:0/1791660699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdab30a0 msgr2=0x5580fdea8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea1000640 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdab30a0 0x5580fdea8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea1000640 1 -- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 msgr2=0x5580fdea28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea1000640 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 0x5580fdea28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea1000640 1 -- 172.21.15.39:0/1791660699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580fdea2e30 con 0x5580fdabb280 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.415+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 0x5580fdea28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:35.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6ea1000640 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fdea23b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001eb0 tx=0x7f6e88003fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d40a264c7c96253 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6e83400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6e83400640 1 -- 172.21.15.39:0/1791660699 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88011af0 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9402c450 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6e83400640 1 -- 172.21.15.39:0/1791660699 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6e88011c90 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6e83400640 1 -- 172.21.15.39:0/1791660699 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e88037050 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6eab02e740 10 client.25342 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.416+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5580fdea87b0 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6e83400640 1 -- 172.21.15.39:0/1791660699 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e9402c450 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6e83400640 1 -- 172.21.15.39:0/1791660699 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6e88019040 con 0x5580fdabb280 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6eab02e740 10 client.25342 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6eab02e740 10 client.25342 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6eab02e740 20 client.25342 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda6fc90 0x5580fdf0f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:35.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6eab02e740 4 client.25342 mds_command: new command op to 24760 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 --> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] -- command(tid 0: {"prefix": "status"}) -- 0x5580fddf7750 con 0x5580fda6fc90 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.417+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda6fc90 0x5580fdf0f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6e83400640 10 client.25342 ms_handle_connect on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6ea2400640 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda6fc90 0x5580fdf0f190 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a27e42359476f6d server_cookie=a79b6f91fd08af51 in_seq=0 out_seq=0 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6e83400640 1 -- 172.21.15.39:0/1791660699 <== mds.1 v2:172.21.15.39:6840/3868264331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5580fddf7750 con 0x5580fda6fc90 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6e83400640 10 client.25342 handle_command_reply: tid=0 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 client.25342 shutdown 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 msgr2=0x5580fdea23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fdea23b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001eb0 tx=0x7f6e88003fe0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 shutdown_connections 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5580fda6fc90 0x5580fdf0f190 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5580fdadbf60 0x5580fdea28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5580fdabb280 0x5580fdea23b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 --2- 172.21.15.39:0/1791660699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5580fdab30a0 0x5580fdea8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6e83400640 0 client.25342 ms_handle_reset on v2:172.21.15.39:6840/3868264331 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.418+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 >> 172.21.15.39:0/1791660699 conn(0x5580fdad0350 msgr2=0x5580fde8c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.419+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 shutdown_connections 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.419+0000 7f6eab02e740 1 -- 172.21.15.39:0/1791660699 wait complete. 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.419+0000 7f6eab02e740 20 client.25342 trim_cache size 0 max 16384 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24760, 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:03:35.424 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 118.09409416299999, 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 188.476164571 2024-09-21T01:03:35.425 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6e840b6560 msgr2=0x7f6e840b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6e840b6560 0x7f6e840b8a70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6e900cd9a0 tx=0x7f6e90031950 comp rx=0 tx=0).stop 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 msgr2=0x7f6e9c1a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c1a08a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e90005940 tx=0x7f6e90004270 comp rx=0 tx=0).stop 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 shutdown_connections 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6e840b6560 0x7f6e840b8a70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e9c114570 0x7f6e9c1a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c078290 0x7f6e9c1a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 --2- 172.21.15.39:0/2103805110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e9c077940 0x7f6e9c1a08a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 >> 172.21.15.39:0/2103805110 conn(0x7f6e9c101680 msgr2=0x7f6e9c116680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 shutdown_connections 2024-09-21T01:03:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:35.420+0000 7f6ea2e00640 1 -- 172.21.15.39:0/2103805110 wait complete. 2024-09-21T01:03:36.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T01:03:36.779 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:37.240 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:36 smithi039 ceph-mon[30415]: pgmap v392: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:37.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:36 smithi150 ceph-mon[34945]: pgmap v392: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:37.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:36 smithi160 ceph-mon[34696]: pgmap v392: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 -- 172.21.15.39:0/2994155537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 msgr2=0x7fb9d0103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2994155537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d0103dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009900 tx=0x7fb9c002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 -- 172.21.15.39:0/2994155537 shutdown_connections 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2994155537 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb9d0109c40 0x7fb9d0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2994155537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d01092f0 0x7fb9d0109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2994155537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d0103dc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 -- 172.21.15.39:0/2994155537 >> 172.21.15.39:0/2994155537 conn(0x7fb9d00ff440 msgr2=0x7fb9d0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 -- 172.21.15.39:0/2994155537 shutdown_connections 2024-09-21T01:03:37.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.419+0000 7fb9d6600640 1 -- 172.21.15.39:0/2994155537 wait complete. 2024-09-21T01:03:37.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.420+0000 7fb9d6600640 1 Processor -- start 2024-09-21T01:03:37.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 -- start start 2024-09-21T01:03:37.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb9d01092f0 0x7fb9d019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d0109c40 0x7fb9d019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d0113900 con 0x7fb9d01039b0 2024-09-21T01:03:37.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d0113780 con 0x7fb9d0109c40 2024-09-21T01:03:37.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d6600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d0113a80 con 0x7fb9d01092f0 2024-09-21T01:03:37.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.421+0000 7fb9d5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51348/0 (socket says 172.21.15.39:51348) 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9c7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb9d01092f0 0x7fb9d019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 -- 172.21.15.39:0/2608132734 learned_addr learned my addr 172.21.15.39:0/2608132734 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d0109c40 0x7fb9d019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 -- 172.21.15.39:0/2608132734 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb9d01092f0 msgr2=0x7fb9d019a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb9d01092f0 0x7fb9d019a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 -- 172.21.15.39:0/2608132734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d0109c40 msgr2=0x7fb9d019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d0109c40 0x7fb9d019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 -- 172.21.15.39:0/2608132734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d01a0260 con 0x7fb9d01039b0 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.422+0000 7fb9d5200640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d019a2f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0005940 tx=0x7fb9c00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.423+0000 7fb9ce000640 1 -- 172.21.15.39:0/2608132734 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c003f030 con 0x7fb9d01039b0 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.423+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d01a04f0 con 0x7fb9d01039b0 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.423+0000 7fb9ce000640 1 -- 172.21.15.39:0/2608132734 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c00044c0 con 0x7fb9d01039b0 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.423+0000 7fb9ce000640 1 -- 172.21.15.39:0/2608132734 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0035070 con 0x7fb9d01039b0 2024-09-21T01:03:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.423+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d01a6cb0 con 0x7fb9d01039b0 2024-09-21T01:03:37.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 Processor -- start 2024-09-21T01:03:37.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 -- start start 2024-09-21T01:03:37.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9ce000640 1 -- 172.21.15.39:0/2608132734 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7fb9c0042030 con 0x7fb9d01039b0 2024-09-21T01:03:37.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 0x555bb9048670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8c790a0 0x555bb9048bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90490f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555bb8fbfd50 con 0x555bb8c790a0 2024-09-21T01:03:37.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555bb8faf180 con 0x555bb8c81280 2024-09-21T01:03:37.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.426+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555bb8fb2f90 con 0x555bb8ca1f60 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8c790a0 0x555bb9048bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 0x555bb9048670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 0x555bb9048670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:47508/0 (socket says 172.21.15.39:47508) 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 -- 172.21.15.39:0/571194873 learned_addr learned my addr 172.21.15.39:0/571194873 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 -- 172.21.15.39:0/571194873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 msgr2=0x555bb90490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 -- 172.21.15.39:0/571194873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8c790a0 msgr2=0x555bb9048bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8c790a0 0x555bb9048bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.427+0000 7fb9d5c00640 1 -- 172.21.15.39:0/571194873 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb8f79a70 con 0x555bb8ca1f60 2024-09-21T01:03:37.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.428+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 0x555bb9048670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009fd0 tx=0x7fb9c80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3de52319a533364e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:37.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.428+0000 7fb9c6000640 1 -- 172.21.15.39:0/571194873 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8004d20 con 0x555bb8ca1f60 2024-09-21T01:03:37.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.428+0000 7fb9c6000640 1 -- 172.21.15.39:0/571194873 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c8004050 con 0x555bb8ca1f60 2024-09-21T01:03:37.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.428+0000 7fb9c6000640 1 -- 172.21.15.39:0/571194873 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8005280 con 0x555bb8ca1f60 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9ce000640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb9b40b6560 0x7fb9b40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 -- 172.21.15.39:0/571194873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 msgr2=0x555bb9048670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 0x555bb9048670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009fd0 tx=0x7fb9c80118d0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9ce000640 1 -- 172.21.15.39:0/2608132734 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb9c0107b40 con 0x7fb9d01039b0 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9d5200640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb9b40b6560 0x7fb9b40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 -- 172.21.15.39:0/571194873 shutdown_connections 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8c790a0 0x555bb9048bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/571194873 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8ca1f60 0x555bb9048670 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 -- 172.21.15.39:0/571194873 >> 172.21.15.39:0/571194873 conn(0x555bb8c96350 msgr2=0x555bb9046560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 -- 172.21.15.39:0/571194873 shutdown_connections 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.429+0000 7fb9de9c3740 1 -- 172.21.15.39:0/571194873 wait complete. 2024-09-21T01:03:37.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.430+0000 7fb9d5200640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb9b40b6560 0x7fb9b40b8a70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x555bb90472f0 tx=0x7fb9c010b620 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.430+0000 7fb9de9c3740 1 -- start start 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.430+0000 7fb9de9c3740 1 Processor -- start 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb90541b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90601a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555bb8e105f0 con 0x555bb8ca1f60 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555bb8f4a460 con 0x555bb8c81280 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555bb9056e50 con 0x555bb8c790a0 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9de9c3740 10 client.0 _ll_register_callbacks cb 0x555bb8f5a210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9c7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51372/0 (socket says 172.21.15.39:51372) 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1540389400 learned_addr learned my addr 172.21.15.39:0/1540389400 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.431+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb90541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1540389400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 msgr2=0x555bb90541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb90541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 msgr2=0x555bb90546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1540389400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb8d0d540 con 0x555bb8ca1f60 2024-09-21T01:03:37.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90601a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80048d0 tx=0x7fb9c8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5804db476d9e5a0d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9bea00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:03:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9bea00640 1 -- 172.21.15.39:0/1540389400 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c80045a0 con 0x555bb8ca1f60 2024-09-21T01:03:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555bb8f4ca70 con 0x555bb8ca1f60 2024-09-21T01:03:37.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.432+0000 7fb9bea00640 1 -- 172.21.15.39:0/1540389400 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb9c8004a40 con 0x555bb8ca1f60 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9bea00640 1 -- 172.21.15.39:0/1540389400 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9c8004c40 con 0x555bb8ca1f60 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 10 client.15345 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555bb8f676f0 con 0x555bb8ca1f60 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9bea00640 1 -- 172.21.15.39:0/1540389400 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555bb8f4ca70 con 0x555bb8ca1f60 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9bea00640 1 -- 172.21.15.39:0/1540389400 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb9c8028450 con 0x555bb8ca1f60 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 10 client.15345 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 10 client.15345 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 20 client.15345 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c608e0 0x555bb90663e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 4 client.15345 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.433+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bb8fc1ca0 con 0x555bb8c608e0 2024-09-21T01:03:37.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.434+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c608e0 0x555bb90663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.435+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c608e0 0x555bb90663e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ab1cb8fb28fe4a9 server_cookie=d0b0790420f36db5 in_seq=0 out_seq=0 2024-09-21T01:03:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.435+0000 7fb9bea00640 10 client.15345 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:37.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9bea00640 1 -- 172.21.15.39:0/1540389400 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555bb8fc1ca0 con 0x555bb8c608e0 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9bea00640 10 client.15345 handle_command_reply: tid=0 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 client.15345 shutdown 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 msgr2=0x555bb90601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90601a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80048d0 tx=0x7fb9c8005bd0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 shutdown_connections 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c608e0 0x555bb90663e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90601a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb90546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1540389400 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb90541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 >> 172.21.15.39:0/1540389400 conn(0x555bb8c96350 msgr2=0x555bb904b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9bea00640 0 client.15345 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 shutdown_connections 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1540389400 wait complete. 2024-09-21T01:03:37.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.436+0000 7fb9de9c3740 20 client.15345 trim_cache size 0 max 16384 2024-09-21T01:03:37.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 Processor -- start 2024-09-21T01:03:37.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 -- start start 2024-09-21T01:03:37.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c790a0 0x555bb90663e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c81280 0x555bb9060110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb9060650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555bb9064720 con 0x555bb8ca1f60 2024-09-21T01:03:37.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555bb9064df0 con 0x555bb8c790a0 2024-09-21T01:03:37.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.456+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555bb90654c0 con 0x555bb8c81280 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb9060650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9c7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c81280 0x555bb9060110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb9060650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51374/0 (socket says 172.21.15.39:51374) 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1344320369 learned_addr learned my addr 172.21.15.39:0/1344320369 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c790a0 0x555bb90663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1344320369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c81280 msgr2=0x555bb9060110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c81280 0x555bb9060110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1344320369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c790a0 msgr2=0x555bb90663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c790a0 0x555bb90663e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 -- 172.21.15.39:0/1344320369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb90524b0 con 0x555bb8ca1f60 2024-09-21T01:03:37.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c81280 0x555bb9060110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:37.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.457+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb9060650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8014040 tx=0x7fb9c8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4116dfc80521512d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:37.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.458+0000 7fb9c5600640 1 -- 172.21.15.39:0/1344320369 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8003dd0 con 0x555bb8ca1f60 2024-09-21T01:03:37.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.458+0000 7fb9c5600640 1 -- 172.21.15.39:0/1344320369 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c8038070 con 0x555bb8ca1f60 2024-09-21T01:03:37.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.458+0000 7fb9c5600640 1 -- 172.21.15.39:0/1344320369 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8004aa0 con 0x555bb8ca1f60 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1344320369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 msgr2=0x555bb9060650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb9060650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8014040 tx=0x7fb9c8009500 comp rx=0 tx=0).stop 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1344320369 shutdown_connections 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb9060650 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c81280 0x555bb9060110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/1344320369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c790a0 0x555bb90663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1344320369 >> 172.21.15.39:0/1344320369 conn(0x555bb8c96350 msgr2=0x555bb904f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:37.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1344320369 shutdown_connections 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 -- 172.21.15.39:0/1344320369 wait complete. 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 -- start start 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.459+0000 7fb9de9c3740 1 Processor -- start 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb906ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb9068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555bb9066260 con 0x555bb8ca1f60 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555bb9065b90 con 0x555bb8c81280 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555bb9066930 con 0x555bb8c790a0 2024-09-21T01:03:37.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9de9c3740 10 client.0 _ll_register_callbacks cb 0x7fb9c8003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9c7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb9068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9d5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9c7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb9068340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:40334/0 (socket says 172.21.15.39:40334) 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.460+0000 7fb9c7e00640 1 -- 172.21.15.39:0/4285384667 learned_addr learned my addr 172.21.15.39:0/4285384667 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb906ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9c7e00640 1 -- 172.21.15.39:0/4285384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 msgr2=0x555bb906ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb906ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9c7e00640 1 -- 172.21.15.39:0/4285384667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 msgr2=0x555bb90688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9c7e00640 1 -- 172.21.15.39:0/4285384667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb9069030 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9c7e00640 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb9068340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8012580 tx=0x7fb9b8012eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8337a73168e0ef8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9bea00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9bea00640 1 -- 172.21.15.39:0/4285384667 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b802b060 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.461+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c8038070 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9bea00640 1 -- 172.21.15.39:0/4285384667 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb9b8004230 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9bea00640 1 -- 172.21.15.39:0/4285384667 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9b8004430 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 10 client.25364 fetch_fsmap learned FSMap version 26 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555bb90d1910 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9bea00640 1 -- 172.21.15.39:0/4285384667 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c8038070 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9bea00640 1 -- 172.21.15.39:0/4285384667 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb9b80047a0 con 0x555bb8c81280 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 10 client.25364 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T01:03:37.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 10 client.25364 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:03:37.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 20 client.25364 populate_metadata read hostname 'smithi039' 2024-09-21T01:03:37.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c35c90 0x555bb90d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:37.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 4 client.25364 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:03:37.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.462+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "status"}) -- 0x555bb8fbd750 con 0x555bb8c35c90 2024-09-21T01:03:37.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.463+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c35c90 0x555bb90d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.463+0000 7fb9d5c00640 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c35c90 0x555bb90d5160 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=396d57160d2c2c90 server_cookie=3a077583974b0155 in_seq=0 out_seq=0 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.463+0000 7fb9bea00640 10 client.25364 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9bea00640 1 -- 172.21.15.39:0/4285384667 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x555bb8fbd750 con 0x555bb8c35c90 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9bea00640 10 client.25364 handle_command_reply: tid=0 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 client.25364 shutdown 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 msgr2=0x555bb9068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb9068340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8012580 tx=0x7fb9b8012eb0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 shutdown_connections 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555bb8c35c90 0x555bb90d5160 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555bb8ca1f60 0x555bb90688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555bb8c81280 0x555bb9068340 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 --2- 172.21.15.39:0/4285384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555bb8c790a0 0x555bb906ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9bea00640 0 client.25364 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 >> 172.21.15.39:0/4285384667 conn(0x555bb8c96350 msgr2=0x555bb90526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 shutdown_connections 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 1 -- 172.21.15.39:0/4285384667 wait complete. 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.464+0000 7fb9de9c3740 20 client.25364 trim_cache size 0 max 16384 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24764, 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:03:37.468 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 172.964655451, 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 187.978917602 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:03:37.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb9b40b6560 msgr2=0x7fb9b40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb9b40b6560 0x7fb9b40b8a70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x555bb90472f0 tx=0x7fb9c010b620 comp rx=0 tx=0).stop 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 msgr2=0x7fb9d019a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d019a2f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0005940 tx=0x7fb9c00040c0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 shutdown_connections 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb9b40b6560 0x7fb9b40b8a70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d0109c40 0x7fb9d019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb9d01092f0 0x7fb9d019a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 --2- 172.21.15.39:0/2608132734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9d01039b0 0x7fb9d019a2f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 >> 172.21.15.39:0/2608132734 conn(0x7fb9d00ff440 msgr2=0x7fb9d0101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 shutdown_connections 2024-09-21T01:03:37.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:37.465+0000 7fb9d6600640 1 -- 172.21.15.39:0/2608132734 wait complete. 2024-09-21T01:03:38.400 INFO:teuthology.run_tasks:Running task kclient... 2024-09-21T01:03:38.419 INFO:tasks.kclient:Mounting kernel clients... 2024-09-21T01:03:38.419 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-21T01:03:38.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:38.420 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-09-21T01:03:38.455 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-21T01:03:38.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:38.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T01:03:38.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:38.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-09-21T01:03:38.906 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-09-21T01:03:38.906 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T01:03:38.906 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T01:03:38.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T01:03:39.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:39 smithi039 ceph-mon[30415]: pgmap v393: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:39.286 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:39 smithi150 ceph-mon[34945]: pgmap v393: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:39.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:39 smithi160 ceph-mon[34696]: pgmap v393: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:03:39.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 -- 172.21.15.39:0/1911284164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 msgr2=0x7f241810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:39.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 --2- 172.21.15.39:0/1911284164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241810b900 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2400009900 tx=0x7f240002a410 comp rx=0 tx=0).stop 2024-09-21T01:03:39.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 -- 172.21.15.39:0/1911284164 shutdown_connections 2024-09-21T01:03:39.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 --2- 172.21.15.39:0/1911284164 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f241810be40 0x7f2418112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:39.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 --2- 172.21.15.39:0/1911284164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241810b900 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:39.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 --2- 172.21.15.39:0/1911284164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 0x7f2418105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:39.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 -- 172.21.15.39:0/1911284164 >> 172.21.15.39:0/1911284164 conn(0x7f2418101680 msgr2=0x7f2418103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:39.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 -- 172.21.15.39:0/1911284164 shutdown_connections 2024-09-21T01:03:39.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.886+0000 7f241ca00640 1 -- 172.21.15.39:0/1911284164 wait complete. 2024-09-21T01:03:39.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.887+0000 7f241ca00640 1 Processor -- start 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.887+0000 7f241ca00640 1 -- start start 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f241ca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 0x7f241819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f241ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f241ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f241810be40 0x7f24181a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f241ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2418115b00 con 0x7f241810b4f0 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f241ca00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2418115980 con 0x7f2418105bb0 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f241ca00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2418115c80 con 0x7f241810be40 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f240fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f2416800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 0x7f241819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f240fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241819cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51404/0 (socket says 172.21.15.39:51404) 2024-09-21T01:03:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f2417200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f241810be40 0x7f24181a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.888+0000 7f240fe00640 1 -- 172.21.15.39:0/2172937341 learned_addr learned my addr 172.21.15.39:0/2172937341 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f240fe00640 1 -- 172.21.15.39:0/2172937341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f241810be40 msgr2=0x7f24181a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f240fe00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f241810be40 0x7f24181a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f240fe00640 1 -- 172.21.15.39:0/2172937341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 msgr2=0x7f241819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f240fe00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 0x7f241819c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f240fe00640 1 -- 172.21.15.39:0/2172937341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24181a24e0 con 0x7f241810b4f0 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f2416800640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 0x7f241819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.889+0000 7f240fe00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241819cb10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f240002a930 tx=0x7f2400031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.890+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240003f030 con 0x7f241810b4f0 2024-09-21T01:03:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.890+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2400004490 con 0x7f241810b4f0 2024-09-21T01:03:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.890+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24181a2770 con 0x7f241810b4f0 2024-09-21T01:03:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.890+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400035070 con 0x7f241810b4f0 2024-09-21T01:03:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.890+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24181a8f30 con 0x7f241810b4f0 2024-09-21T01:03:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.891+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2418106e30 con 0x7f241810b4f0 2024-09-21T01:03:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.893+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f2400042030 con 0x7f241810b4f0 2024-09-21T01:03:39.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.895+0000 7f240f400640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f23fc0b65e0 0x7f23fc0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:39.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.895+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2400107ac0 con 0x7f241810b4f0 2024-09-21T01:03:39.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.895+0000 7f2416800640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f23fc0b65e0 0x7f23fc0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:39.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.896+0000 7f2416800640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f23fc0b65e0 0x7f23fc0b8af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2408009fc0 tx=0x7f2408005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:39.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:39.899+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f24000d2230 con 0x7f241810b4f0 2024-09-21T01:03:40.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.139+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f241810dc30 con 0x7f241810b4f0 2024-09-21T01:03:40.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.140+0000 7f240f400640 1 -- 172.21.15.39:0/2172937341 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+88 (secure 0 0 0) 0x7f24000d7170 con 0x7f241810b4f0 2024-09-21T01:03:40.141 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-21T01:03:40.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f23fc0b65e0 msgr2=0x7f23fc0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:40.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f23fc0b65e0 0x7f23fc0b8af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2408009fc0 tx=0x7f2408005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 msgr2=0x7f241819cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241819cb10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f240002a930 tx=0x7f2400031a20 comp rx=0 tx=0).stop 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 shutdown_connections 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f241810be40 0x7f24181a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f241810b4f0 0x7f241819cb10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f23fc0b65e0 0x7f23fc0b8af0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:40.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 --2- 172.21.15.39:0/2172937341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2418105bb0 0x7f241819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:40.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.142+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 >> 172.21.15.39:0/2172937341 conn(0x7f2418101680 msgr2=0x7f2418103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:40.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.143+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 shutdown_connections 2024-09-21T01:03:40.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:40.143+0000 7f241ca00640 1 -- 172.21.15.39:0/2172937341 wait complete. 2024-09-21T01:03:40.916 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T01:03:40.917 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-21T01:03:40.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:40.917 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-09-21T01:03:40.953 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T01:03:40.953 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:03:40.954 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 41933sec preferred_lft 41933sec 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:03:40.955 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-09-21T01:03:40.956 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T01:03:40.956 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:03:41.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:03:41.142 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:41 smithi039 ceph-mon[30415]: pgmap v394: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:41.142 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:41 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2172937341' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:03:41.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:41.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:41.183 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T01:03:41.239 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-09-21T01:03:41.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:41.242 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-09-21T01:03:41.279 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-09-21T01:03:41.279 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T01:03:41.279 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T01:03:41.279 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T01:03:41.279 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T01:03:41.280 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:03:41.280 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:03:41.280 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T01:03:41.280 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T01:03:41.280 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T01:03:41.280 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:03:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:41 smithi150 ceph-mon[34945]: pgmap v394: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:41 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2172937341' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:03:41.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:41 smithi160 ceph-mon[34696]: pgmap v394: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:03:41.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:41 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2172937341' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:03:41.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:03:41.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:41.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:41.487 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-09-21T01:03:41.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:41.524 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-09-21T01:03:41.599 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:03:41.600 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:03:41.600 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:03:41.600 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-21T01:03:41.600 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:03:41.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:03:41.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:41.778 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-21T01:03:41.778 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:03:41.778 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:03:41.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-21T01:03:41.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T01:03:41.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-21T01:03:41.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-21T01:03:41.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-21T01:03:41.779 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:03:41.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:03:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:41.999 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:03:41.999 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-09-21T01:03:41.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-09-21T01:03:41.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T01:03:41.999 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-09-21T01:03:42.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:03:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:42.167 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:03:42.168 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-21T01:03:42.203 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-21T01:03:42.203 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:03:42.203 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-21T01:03:42.279 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-21T01:03:42.279 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-21T01:03:42.279 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/c561b996-0695-4404-89f5-09c45de940bd 2024-09-21T01:03:42.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:42.279 DEBUG:teuthology.orchestra.run.smithi039:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/c561b996-0695-4404-89f5-09c45de940bd /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-09-21T01:03:42.478 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-09-21T01:03:42.478 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-09-21T01:03:43.184 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-09-21T01:03:43.185 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-09-21T01:03:43.185 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.39:6789,172.21.15.150:6789,172.21.15.160:6789" 2024-09-21T01:03:43.185 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: trying mount with new device syntax: 0@61abfb08-77b3-11ef-baf5-efdc52797490.cephfs=/volumes/qa/sv_1/c561b996-0695-4404-89f5-09c45de940bd 2024-09-21T01:03:43.185 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.39:6789/172.21.15.150:6789/172.21.15.160:6789" will pass to kernel 2024-09-21T01:03:43.187 INFO:teuthology.orchestra.run.smithi039.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-21T01:03:43.187 INFO:teuthology.orchestra.run.smithi039.stdout: You just mounted a file system that supports labels which does not 2024-09-21T01:03:43.187 INFO:teuthology.orchestra.run.smithi039.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-21T01:03:43.187 INFO:teuthology.orchestra.run.smithi039.stdout: applications will generate AVC messages and not be allowed access to 2024-09-21T01:03:43.187 INFO:teuthology.orchestra.run.smithi039.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-21T01:03:43.190 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-21T01:03:43.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:03:43.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-21T01:03:43.253 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-21T01:03:43.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:43 smithi150 ceph-mon[34945]: pgmap v395: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:43.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T01:03:43.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:43.345 INFO:teuthology.orchestra.run.smithi039.stdout:client25373 2024-09-21T01:03:43.345 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:03:43.346 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:03:43.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:43 smithi160 ceph-mon[34696]: pgmap v395: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:43.382 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-21T01:03:43.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:43 smithi039 ceph-mon[30415]: pgmap v395: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:03:43.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T01:03:43.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:43.494 INFO:teuthology.orchestra.run.smithi039.stdout:client25373 2024-09-21T01:03:43.494 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-21T01:03:43.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T01:03:43.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:43.585 INFO:teuthology.orchestra.run.smithi039.stdout:client25373 2024-09-21T01:03:43.586 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/61abfb08-77b3-11ef-baf5-efdc52797490.client25373/status') 2024-09-21T01:03:43.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:03:43.686 INFO:teuthology.orchestra.run.smithi039.stdout:instance: client.25373 (3)172.21.15.39:0/4144412853 2024-09-21T01:03:43.686 INFO:teuthology.orchestra.run.smithi039.stdout:blocklisted: false 2024-09-21T01:03:43.686 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records in 2024-09-21T01:03:43.686 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records out 2024-09-21T01:03:43.686 INFO:teuthology.orchestra.run.smithi039.stderr:71 bytes copied, 8.6742e-05 s, 819 kB/s 2024-09-21T01:03:43.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:03:43.697 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T01:03:43.713 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T01:03:43.713 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:03:43.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 -- 172.21.15.39:0/2005206836 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 msgr2=0x7f00641100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 --2- 172.21.15.39:0/2005206836 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 0x7f00641100b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f005c009ca0 tx=0x7f005c02a8b0 comp rx=0 tx=0).stop 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 -- 172.21.15.39:0/2005206836 shutdown_connections 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 --2- 172.21.15.39:0/2005206836 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 0x7f00641100b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 --2- 172.21.15.39:0/2005206836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00641050b0 0x7f006410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 --2- 172.21.15.39:0/2005206836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0064104760 0x7f0064104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 -- 172.21.15.39:0/2005206836 >> 172.21.15.39:0/2005206836 conn(0x7f00640fe230 msgr2=0x7f0064100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:43.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 -- 172.21.15.39:0/2005206836 shutdown_connections 2024-09-21T01:03:43.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.933+0000 7f0063e00640 1 -- 172.21.15.39:0/2005206836 wait complete. 2024-09-21T01:03:43.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.934+0000 7f0063e00640 1 Processor -- start 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.934+0000 7f0063e00640 1 -- start start 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0064104760 0x7f006419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 0x7f006419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 0x7f006419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f00641138a0 con 0x7f00641050b0 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0064113720 con 0x7f0064104760 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0064113a20 con 0x7f006410dbc0 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f005be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 0x7f006419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f005be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 0x7f006419a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57430/0 (socket says 172.21.15.39:57430) 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f005be00640 1 -- 172.21.15.39:0/4020631656 learned_addr learned my addr 172.21.15.39:0/4020631656 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0062a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0064104760 0x7f006419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.935+0000 7f0063400640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 0x7f006419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:43.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005be00640 1 -- 172.21.15.39:0/4020631656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 msgr2=0x7f006419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005be00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 0x7f006419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005be00640 1 -- 172.21.15.39:0/4020631656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0064104760 msgr2=0x7f006419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005be00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0064104760 0x7f006419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005be00640 1 -- 172.21.15.39:0/4020631656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00641a0170 con 0x7f00641050b0 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f0062a00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0064104760 0x7f006419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005be00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 0x7f006419a7e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f004c00e9c0 tx=0x7f004c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c00ccc0 con 0x7f00641050b0 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00641a0460 con 0x7f00641050b0 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f004c0054d0 con 0x7f00641050b0 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c010430 con 0x7f00641050b0 2024-09-21T01:03:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.936+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00641a6bf0 con 0x7f00641050b0 2024-09-21T01:03:43.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.937+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0064104b70 con 0x7f00641050b0 2024-09-21T01:03:43.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.940+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f004c005020 con 0x7f00641050b0 2024-09-21T01:03:43.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.941+0000 7f005b400640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00480b6560 0x7f00480b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.942+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f004c0e7d80 con 0x7f00641050b0 2024-09-21T01:03:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.942+0000 7f0062a00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00480b6560 0x7f00480b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.942+0000 7f0062a00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00480b6560 0x7f00480b8a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f00540095f0 tx=0x7f0054005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:43.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:43.944+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f004c056710 con 0x7f00641050b0 2024-09-21T01:03:44.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:44.183+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f00641a6f50 con 0x7f00641050b0 2024-09-21T01:03:45.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.028+0000 7f005b400640 1 -- 172.21.15.39:0/4020631656 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7f004c0ae110 con 0x7f00641050b0 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00480b6560 msgr2=0x7f00480b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00480b6560 0x7f00480b8a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f00540095f0 tx=0x7f0054005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 msgr2=0x7f006419a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 0x7f006419a7e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f004c00e9c0 tx=0x7f004c00ee80 comp rx=0 tx=0).stop 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 shutdown_connections 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f006410dbc0 0x7f006419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00641050b0 0x7f006419a7e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f00480b6560 0x7f00480b8a70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.031+0000 7f0063e00640 1 --2- 172.21.15.39:0/4020631656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0064104760 0x7f006419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.032+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 >> 172.21.15.39:0/4020631656 conn(0x7f00640fe230 msgr2=0x7f00640ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:45.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.032+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 shutdown_connections 2024-09-21T01:03:45.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.032+0000 7f0063e00640 1 -- 172.21.15.39:0/4020631656 wait complete. 2024-09-21T01:03:45.062 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-21T01:03:45.074 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T01:03:45.102 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T01:03:45.103 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:03:45.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-09-21T01:03:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:45 smithi150 ceph-mon[34945]: pgmap v396: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4020631656' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T01:03:45.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 -- 172.21.15.39:0/715162496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 msgr2=0x7fa1b4075b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 --2- 172.21.15.39:0/715162496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b4075b60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009900 tx=0x7fa1a802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 -- 172.21.15.39:0/715162496 shutdown_connections 2024-09-21T01:03:45.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 --2- 172.21.15.39:0/715162496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa1b4110080 0x7fa1b41124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 --2- 172.21.15.39:0/715162496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b40760a0 0x7fa1b410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 --2- 172.21.15.39:0/715162496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b4075b60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.328+0000 7fa1bc600640 1 -- 172.21.15.39:0/715162496 >> 172.21.15.39:0/715162496 conn(0x7fa1b40ff510 msgr2=0x7fa1b4101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:45.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.329+0000 7fa1bc600640 1 -- 172.21.15.39:0/715162496 shutdown_connections 2024-09-21T01:03:45.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.329+0000 7fa1bc600640 1 -- 172.21.15.39:0/715162496 wait complete. 2024-09-21T01:03:45.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 Processor -- start 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 -- start start 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa1b40760a0 0x7fa1b419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b4110080 0x7fa1b41a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b4115c40 con 0x7fa1b4075750 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b4115ac0 con 0x7fa1b4110080 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1bc600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b4115dc0 con 0x7fa1b40760a0 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1b9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1b9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57438/0 (socket says 172.21.15.39:57438) 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.330+0000 7fa1b9c00640 1 -- 172.21.15.39:0/1504684931 learned_addr learned my addr 172.21.15.39:0/1504684931 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1ba600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b4110080 0x7fa1b41a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9200640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa1b40760a0 0x7fa1b419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9c00640 1 -- 172.21.15.39:0/1504684931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa1b40760a0 msgr2=0x7fa1b419cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9c00640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa1b40760a0 0x7fa1b419cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9c00640 1 -- 172.21.15.39:0/1504684931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b4110080 msgr2=0x7fa1b41a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9c00640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b4110080 0x7fa1b41a1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9c00640 1 -- 172.21.15.39:0/1504684931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b41a2590 con 0x7fa1b4075750 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b9c00640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b419c660 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8005ab0 tx=0x7fa1a8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a803f070 con 0x7fa1b4075750 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1a8031b60 con 0x7fa1b4075750 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.331+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b41a2820 con 0x7fa1b4075750 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.332+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8035030 con 0x7fa1b4075750 2024-09-21T01:03:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.332+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b41a9010 con 0x7fa1b4075750 2024-09-21T01:03:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.333+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b4075b60 con 0x7fa1b4075750 2024-09-21T01:03:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.335+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7fa1a8042070 con 0x7fa1b4075750 2024-09-21T01:03:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.336+0000 7fa1b2000640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa19c0b65e0 0x7fa19c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.337+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa1a8107ed0 con 0x7fa1b4075750 2024-09-21T01:03:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.337+0000 7fa1b9200640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa19c0b65e0 0x7fa19c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.337+0000 7fa1b9200640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa19c0b65e0 0x7fa19c0b8af0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4112420 tx=0x7fa1a0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:45.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.341+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa1a80d2640 con 0x7fa1b4075750 2024-09-21T01:03:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:45 smithi160 ceph-mon[34696]: pgmap v396: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4020631656' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T01:03:45.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:45 smithi039 ceph-mon[30415]: pgmap v396: 129 pgs: 129 active+clean; 649 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:03:45.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4020631656' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T01:03:45.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.561+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa1b410a2a0 con 0x7fa19c0b65e0 2024-09-21T01:03:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.743+0000 7fa1b2000640 1 -- 172.21.15.39:0/1504684931 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa1b410a2a0 con 0x7fa19c0b65e0 2024-09-21T01:03:45.744 INFO:teuthology.orchestra.run.smithi039.stdout:Schedule set for path / 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa19c0b65e0 msgr2=0x7fa19c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa19c0b65e0 0x7fa19c0b8af0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4112420 tx=0x7fa1a0005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 msgr2=0x7fa1b419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b419c660 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8005ab0 tx=0x7fa1a8003c10 comp rx=0 tx=0).stop 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 shutdown_connections 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa19c0b65e0 0x7fa19c0b8af0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b4110080 0x7fa1b41a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa1b40760a0 0x7fa1b419cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 --2- 172.21.15.39:0/1504684931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b4075750 0x7fa1b419c660 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 >> 172.21.15.39:0/1504684931 conn(0x7fa1b40ff510 msgr2=0x7fa1b4109b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.745+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 shutdown_connections 2024-09-21T01:03:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.746+0000 7fa1bc600640 1 -- 172.21.15.39:0/1504684931 wait complete. 2024-09-21T01:03:45.768 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-09-21T01:03:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 -- 172.21.15.39:0/1045466481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109ba0 msgr2=0x7f5c6c1103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 --2- 172.21.15.39:0/1045466481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c1103d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009ca0 tx=0x7f5c6402a8b0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 -- 172.21.15.39:0/1045466481 shutdown_connections 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 --2- 172.21.15.39:0/1045466481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c1103d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 --2- 172.21.15.39:0/1045466481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109250 0x7f5c6c109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 --2- 172.21.15.39:0/1045466481 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 -- 172.21.15.39:0/1045466481 >> 172.21.15.39:0/1045466481 conn(0x7f5c6c0ff320 msgr2=0x7f5c6c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 -- 172.21.15.39:0/1045466481 shutdown_connections 2024-09-21T01:03:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.991+0000 7f5c71c00640 1 -- 172.21.15.39:0/1045466481 wait complete. 2024-09-21T01:03:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.992+0000 7f5c71c00640 1 Processor -- start 2024-09-21T01:03:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.992+0000 7f5c71c00640 1 -- start start 2024-09-21T01:03:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109250 0x7f5c6c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1138b0 con 0x7f5c6c109250 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c113730 con 0x7f5c6c109ba0 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c113a30 con 0x7f5c6c103920 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c63e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109250 0x7f5c6c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43384/0 (socket says 172.21.15.39:43384) 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c6be00640 1 -- 172.21.15.39:0/174991262 learned_addr learned my addr 172.21.15.39:0/174991262 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71200640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c63e00640 1 -- 172.21.15.39:0/174991262 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 msgr2=0x7f5c6c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c63e00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c63e00640 1 -- 172.21.15.39:0/174991262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109ba0 msgr2=0x7f5c6c19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c63e00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c63e00640 1 -- 172.21.15.39:0/174991262 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c1a01c0 con 0x7f5c6c109250 2024-09-21T01:03:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c6be00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.993+0000 7f5c71200640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.994+0000 7f5c63e00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109250 0x7f5c6c19a830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c06c820 tx=0x7f5c5400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.994+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400cc40 con 0x7f5c6c109250 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.994+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c5400cde0 con 0x7f5c6c109250 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.994+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54018780 con 0x7f5c6c109250 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.994+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c1a04b0 con 0x7f5c6c109250 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.994+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c1a6c40 con 0x7f5c6c109250 2024-09-21T01:03:45.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.995+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6c103d30 con 0x7f5c6c109250 2024-09-21T01:03:45.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.997+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f5c54010a90 con 0x7f5c6c109250 2024-09-21T01:03:46.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.999+0000 7f5c69600640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5c500b65e0 0x7f5c500b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:46.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.999+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c540e7f40 con 0x7f5c6c109250 2024-09-21T01:03:46.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:45.999+0000 7f5c6be00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5c500b65e0 0x7f5c500b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:46.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.000+0000 7f5c6be00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5c500b65e0 0x7f5c500b8af0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0095f0 tx=0x7f5c5c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:46.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.001+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5c540ae330 con 0x7f5c6c109250 2024-09-21T01:03:46.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.053+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f5c540a9ea0 con 0x7f5c6c109250 2024-09-21T01:03:46.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.232+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f5c6c063800 con 0x7f5c500b65e0 2024-09-21T01:03:46.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.307+0000 7f5c69600640 1 -- 172.21.15.39:0/174991262 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f5c6c063800 con 0x7f5c500b65e0 2024-09-21T01:03:46.308 INFO:teuthology.orchestra.run.smithi039.stdout:Retention added to path / 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5c500b65e0 msgr2=0x7f5c500b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5c500b65e0 0x7f5c500b8af0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0095f0 tx=0x7f5c5c005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109250 msgr2=0x7f5c6c19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109250 0x7f5c6c19a830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c06c820 tx=0x7f5c5400ee50 comp rx=0 tx=0).stop 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 shutdown_connections 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5c500b65e0 0x7f5c500b8af0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c6c109ba0 0x7f5c6c19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c6c109250 0x7f5c6c19a830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 --2- 172.21.15.39:0/174991262 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c6c103920 0x7f5c6c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.309+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 >> 172.21.15.39:0/174991262 conn(0x7f5c6c0ff320 msgr2=0x7f5c6c108800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.310+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 shutdown_connections 2024-09-21T01:03:46.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.310+0000 7f5c71c00640 1 -- 172.21.15.39:0/174991262 wait complete. 2024-09-21T01:03:46.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:46 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4020631656' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T01:03:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:46 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:03:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:46 smithi150 ceph-mon[34945]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:46.334 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:46 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4020631656' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T01:03:46.334 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:46 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:03:46.334 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:46 smithi039 ceph-mon[30415]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:46.336 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-21T01:03:46.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:46 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4020631656' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T01:03:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:46 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:03:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:46 smithi160 ceph-mon[34696]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:46.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 -- 172.21.15.39:0/3684209539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 msgr2=0x7f3f9c104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 --2- 172.21.15.39:0/3684209539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c104b50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009900 tx=0x7f3f8c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 -- 172.21.15.39:0/3684209539 shutdown_connections 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 --2- 172.21.15.39:0/3684209539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c10dba0 0x7f3f9c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 --2- 172.21.15.39:0/3684209539 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f9c105090 0x7f3f9c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 --2- 172.21.15.39:0/3684209539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c104b50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 -- 172.21.15.39:0/3684209539 >> 172.21.15.39:0/3684209539 conn(0x7f3f9c0fe230 msgr2=0x7f3f9c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 -- 172.21.15.39:0/3684209539 shutdown_connections 2024-09-21T01:03:46.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.561+0000 7f3f9b200640 1 -- 172.21.15.39:0/3684209539 wait complete. 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.562+0000 7f3f9b200640 1 Processor -- start 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.562+0000 7f3f9b200640 1 -- start start 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c105090 0x7f3f9c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f9c10dba0 0x7f3f9c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c113850 con 0x7f3f9c104740 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c1136d0 con 0x7f3f9c105090 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c1139d0 con 0x7f3f9c10dba0 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f99e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f99e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57496/0 (socket says 172.21.15.39:57496) 2024-09-21T01:03:46.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f99e00640 1 -- 172.21.15.39:0/1514549906 learned_addr learned my addr 172.21.15.39:0/1514549906 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f93e00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c105090 0x7f3f9c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.563+0000 7f3f9a800640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f9c10dba0 0x7f3f9c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f99e00640 1 -- 172.21.15.39:0/1514549906 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f9c10dba0 msgr2=0x7f3f9c19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f99e00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f9c10dba0 0x7f3f9c19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f99e00640 1 -- 172.21.15.39:0/1514549906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c105090 msgr2=0x7f3f9c19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f99e00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c105090 0x7f3f9c19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f99e00640 1 -- 172.21.15.39:0/1514549906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c1a0260 con 0x7f3f9c104740 2024-09-21T01:03:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f99e00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c19a330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c005ab0 tx=0x7f3f8c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c03f070 con 0x7f3f9c104740 2024-09-21T01:03:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9c1a04f0 con 0x7f3f9c104740 2024-09-21T01:03:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8c031d60 con 0x7f3f9c104740 2024-09-21T01:03:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c035030 con 0x7f3f9c104740 2024-09-21T01:03:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.564+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9c1a6ce0 con 0x7f3f9c104740 2024-09-21T01:03:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.565+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f9c104b50 con 0x7f3f9c104740 2024-09-21T01:03:46.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.568+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f3f8c042070 con 0x7f3f9c104740 2024-09-21T01:03:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.569+0000 7f3f92a00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.569+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3f8c1081a0 con 0x7f3f9c104740 2024-09-21T01:03:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.569+0000 7f3f93e00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.570+0000 7f3f93e00640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f840095f0 tx=0x7f3f84005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:46.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.573+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3f8c0d27b0 con 0x7f3f9c104740 2024-09-21T01:03:46.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.798+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3f9c107da0 con 0x7f3f800b65f0 2024-09-21T01:03:46.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.806+0000 7f3f92a00640 1 -- 172.21.15.39:0/1514549906 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f3f9c107da0 con 0x7f3f800b65f0 2024-09-21T01:03:46.807 INFO:teuthology.orchestra.run.smithi039.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-21T00:00:00", "created": "2024-09-21T01:03:45", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 msgr2=0x7f3f800b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f840095f0 tx=0x7f3f84005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 msgr2=0x7f3f9c19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c19a330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c005ab0 tx=0x7f3f8c003c80 comp rx=0 tx=0).stop 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 shutdown_connections 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f9c10dba0 0x7f3f9c19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c105090 0x7f3f9c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 --2- 172.21.15.39:0/1514549906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f9c104740 0x7f3f9c19a330 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.808+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 >> 172.21.15.39:0/1514549906 conn(0x7f3f9c0fe230 msgr2=0x7f3f9c107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.809+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 shutdown_connections 2024-09-21T01:03:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:46.809+0000 7f3f9b200640 1 -- 172.21.15.39:0/1514549906 wait complete. 2024-09-21T01:03:46.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 -- 172.21.15.39:0/421278913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 msgr2=0x7fcc38109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 --2- 172.21.15.39:0/421278913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 0x7fcc38109650 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009900 tx=0x7fcc2402a410 comp rx=0 tx=0).stop 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 -- 172.21.15.39:0/421278913 shutdown_connections 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 --2- 172.21.15.39:0/421278913 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38109b90 0x7fcc381103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 --2- 172.21.15.39:0/421278913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 0x7fcc38109650 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 --2- 172.21.15.39:0/421278913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38103910 0x7fcc38103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 -- 172.21.15.39:0/421278913 >> 172.21.15.39:0/421278913 conn(0x7fcc380ff330 msgr2=0x7fcc38101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 -- 172.21.15.39:0/421278913 shutdown_connections 2024-09-21T01:03:47.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.054+0000 7fcc3e200640 1 -- 172.21.15.39:0/421278913 wait complete. 2024-09-21T01:03:47.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.055+0000 7fcc3e200640 1 Processor -- start 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 -- start start 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 0x7fcc38075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 0x7fcc38075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38109b90 0x7fcc38076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcc38113850 con 0x7fcc38109240 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381136d0 con 0x7fcc38109b90 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3e200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381139d0 con 0x7fcc38103910 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3ce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 0x7fcc38075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3ce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 0x7fcc38075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43430/0 (socket says 172.21.15.39:43430) 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc3ce00640 1 -- 172.21.15.39:0/2749053371 learned_addr learned my addr 172.21.15.39:0/2749053371 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.056+0000 7fcc37e00640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 0x7fcc38075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3d800640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38109b90 0x7fcc38076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:47.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3ce00640 1 -- 172.21.15.39:0/2749053371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38109b90 msgr2=0x7fcc38076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3ce00640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38109b90 0x7fcc38076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3ce00640 1 -- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 msgr2=0x7fcc38075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3ce00640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 0x7fcc38075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3ce00640 1 -- 172.21.15.39:0/2749053371 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc381a8520 con 0x7fcc38103910 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3d800640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38109b90 0x7fcc38076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3ce00640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 0x7fcc38075710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00d950 tx=0x7fcc2c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c017030 con 0x7fcc38103910 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc381a8810 con 0x7fcc38103910 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2c004020 con 0x7fcc38103910 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c0047a0 con 0x7fcc38103910 2024-09-21T01:03:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.057+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc381a8d80 con 0x7fcc38103910 2024-09-21T01:03:47.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.058+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc38103d20 con 0x7fcc38103910 2024-09-21T01:03:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.061+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fcc2c009a30 con 0x7fcc38103910 2024-09-21T01:03:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.062+0000 7fcc35600640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcc200b66a0 0x7fcc200b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.062+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc2c0e7500 con 0x7fcc38103910 2024-09-21T01:03:47.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.062+0000 7fcc37e00640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcc200b66a0 0x7fcc200b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:47.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.063+0000 7fcc37e00640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcc200b66a0 0x7fcc200b8bb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009230 tx=0x7fcc24031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.064+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcc2c0ad7b0 con 0x7fcc38103910 2024-09-21T01:03:47.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:47 smithi039 ceph-mon[30415]: pgmap v397: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-21T01:03:47.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:47 smithi039 ceph-mon[30415]: mgrmap e23: x(active, since 10m), standbys: z, y 2024-09-21T01:03:47.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:47 smithi039 ceph-mon[30415]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:47.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.284+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fcc38063800 con 0x7fcc200b66a0 2024-09-21T01:03:47.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.292+0000 7fcc35600640 1 -- 172.21.15.39:0/2749053371 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fcc38063800 con 0x7fcc200b66a0 2024-09-21T01:03:47.293 INFO:teuthology.orchestra.run.smithi039.stdout:/ 1m 6m3h 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.294+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcc200b66a0 msgr2=0x7fcc200b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.294+0000 7fcc3e200640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcc200b66a0 0x7fcc200b8bb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009230 tx=0x7fcc24031400 comp rx=0 tx=0).stop 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.294+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 msgr2=0x7fcc38075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.294+0000 7fcc3e200640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 0x7fcc38075710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00d950 tx=0x7fcc2c00de10 comp rx=0 tx=0).stop 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.294+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 shutdown_connections 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcc200b66a0 0x7fcc200b8bb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc38109b90 0x7fcc38076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc38109240 0x7fcc38075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 --2- 172.21.15.39:0/2749053371 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcc38103910 0x7fcc38075710 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:47.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 >> 172.21.15.39:0/2749053371 conn(0x7fcc380ff330 msgr2=0x7fcc38101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:47.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 shutdown_connections 2024-09-21T01:03:47.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:47.295+0000 7fcc3e200640 1 -- 172.21.15.39:0/2749053371 wait complete. 2024-09-21T01:03:47.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:47 smithi150 ceph-mon[34945]: pgmap v397: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-21T01:03:47.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:47 smithi150 ceph-mon[34945]: mgrmap e23: x(active, since 10m), standbys: z, y 2024-09-21T01:03:47.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:47 smithi150 ceph-mon[34945]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:47.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-09-21T01:03:47.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:47 smithi160 ceph-mon[34696]: pgmap v397: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-21T01:03:47.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:47 smithi160 ceph-mon[34696]: mgrmap e23: x(active, since 10m), standbys: z, y 2024-09-21T01:03:47.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:47 smithi160 ceph-mon[34696]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:47.380 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-09-21T01:03:47.390 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-21T01:03:47.415 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T01:03:47.415 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:03:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:48.267 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:48 smithi039 ceph-mon[30415]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:48.267 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:48 smithi039 ceph-mon[30415]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:48 smithi150 ceph-mon[34945]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:48 smithi150 ceph-mon[34945]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:48 smithi160 ceph-mon[34696]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:48 smithi160 ceph-mon[34696]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:48.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 -- 172.21.15.39:0/3123947668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c109c40 msgr2=0x7ff04c110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:48.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 --2- 172.21.15.39:0/3123947668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c109c40 0x7ff04c110470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff044009ca0 tx=0x7ff04402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 -- 172.21.15.39:0/3123947668 shutdown_connections 2024-09-21T01:03:48.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 --2- 172.21.15.39:0/3123947668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c109c40 0x7ff04c110470 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 --2- 172.21.15.39:0/3123947668 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1092f0 0x7ff04c109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 --2- 172.21.15.39:0/3123947668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff04c1039b0 0x7ff04c103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.445+0000 7ff051c00640 1 -- 172.21.15.39:0/3123947668 >> 172.21.15.39:0/3123947668 conn(0x7ff04c0ff440 msgr2=0x7ff04c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:48.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.446+0000 7ff051c00640 1 -- 172.21.15.39:0/3123947668 shutdown_connections 2024-09-21T01:03:48.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.446+0000 7ff051c00640 1 -- 172.21.15.39:0/3123947668 wait complete. 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 Processor -- start 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 -- start start 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1039b0 0x7ff04c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c1092f0 0x7ff04c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff04c109c40 0x7ff04c1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c113900 con 0x7ff04c1092f0 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c113780 con 0x7ff04c109c40 2024-09-21T01:03:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff051c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c113a80 con 0x7ff04c1039b0 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff04b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c1092f0 0x7ff04c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff04b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c1092f0 0x7ff04c19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57552/0 (socket says 172.21.15.39:57552) 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff04b400640 1 -- 172.21.15.39:0/1858610288 learned_addr learned my addr 172.21.15.39:0/1858610288 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.447+0000 7ff04be00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1039b0 0x7ff04c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff051200640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff04c109c40 0x7ff04c1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff04be00640 1 -- 172.21.15.39:0/1858610288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff04c109c40 msgr2=0x7ff04c1a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff04be00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff04c109c40 0x7ff04c1a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff04be00640 1 -- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c1092f0 msgr2=0x7ff04c19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff04be00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c1092f0 0x7ff04c19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff04be00640 1 -- 172.21.15.39:0/1858610288 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c1a2420 con 0x7ff04c1039b0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.448+0000 7ff04be00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1039b0 0x7ff04c19c4b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff03c005fd0 tx=0x7ff03c00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.449+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c009c30 con 0x7ff04c1039b0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.449+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c1a2710 con 0x7ff04c1039b0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.449+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff03c009dd0 con 0x7ff04c1039b0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.449+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c003e20 con 0x7ff04c1039b0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.449+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04c1a8e70 con 0x7ff04c1039b0 2024-09-21T01:03:48.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.450+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff04c103dc0 con 0x7ff04c1039b0 2024-09-21T01:03:48.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.453+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7ff03c003fe0 con 0x7ff04c1039b0 2024-09-21T01:03:48.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.454+0000 7ff048c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0300b65f0 0x7ff0300b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:48.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.454+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff03c0e7530 con 0x7ff04c1039b0 2024-09-21T01:03:48.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.454+0000 7ff04b400640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0300b65f0 0x7ff0300b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:48.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.455+0000 7ff04b400640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0300b65f0 0x7ff0300b8b00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff04c109b10 tx=0x7ff03800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:48.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.455+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff03c017020 con 0x7ff04c1039b0 2024-09-21T01:03:48.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.656+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff04c10f740 con 0x7ff0300b65f0 2024-09-21T01:03:48.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.664+0000 7ff048c00640 1 -- 172.21.15.39:0/1858610288 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340531 (secure 0 0 0) 0x7ff04c10f740 con 0x7ff0300b65f0 2024-09-21T01:03:48.669 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:48.671 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:03:48.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0300b65f0 msgr2=0x7ff0300b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:48.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0300b65f0 0x7ff0300b8b00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff04c109b10 tx=0x7ff03800b410 comp rx=0 tx=0).stop 2024-09-21T01:03:48.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1039b0 msgr2=0x7ff04c19c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:48.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1039b0 0x7ff04c19c4b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff03c005fd0 tx=0x7ff03c00dde0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 shutdown_connections 2024-09-21T01:03:48.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0300b65f0 0x7ff0300b8b00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff04c109c40 0x7ff04c1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff04c1092f0 0x7ff04c19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 --2- 172.21.15.39:0/1858610288 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff04c1039b0 0x7ff04c19c4b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:48.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 >> 172.21.15.39:0/1858610288 conn(0x7ff04c0ff440 msgr2=0x7ff04c108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:48.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 shutdown_connections 2024-09-21T01:03:48.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:48.670+0000 7ff051c00640 1 -- 172.21.15.39:0/1858610288 wait complete. 2024-09-21T01:03:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:49 smithi150 ceph-mon[34945]: pgmap v398: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:49 smithi150 ceph-mon[34945]: from='client.25408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:49.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:49 smithi160 ceph-mon[34696]: pgmap v398: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:49.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:49 smithi160 ceph-mon[34696]: from='client.25408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:49 smithi039 ceph-mon[30415]: pgmap v398: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:49 smithi039 ceph-mon[30415]: from='client.25408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:49.844 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-21T01:03:47.940156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1456246,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1198,"num_read_kb":842,"num_write":537,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":898,"ondisk_log_size":898,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373532,"kb_used_data":15844,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124568292,"statfs":{"total":1151940427776,"available":1151557931008,"internally_reserved":0,"allocated":16224256,"data_stored":9967486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":791963,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":410,"num_read_kb":339,"num_write":233,"num_write_kb":2099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003696"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489181+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:02:10.489181+0000","last_peered":"2024-09-21T01:02:10.489181+0000","last_clean":"2024-09-21T01:02:10.489181+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:02:10.489181+0000","last_undegraded":"2024-09-21T01:02:10.489181+0000","last_fullsized":"2024-09-21T01:02:10.489181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296157+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:03:08.296157+0000","last_peered":"2024-09-21T01:03:08.296157+0000","last_clean":"2024-09-21T01:03:08.296157+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:03:08.296157+0000","last_undegraded":"2024-09-21T01:03:08.296157+0000","last_fullsized":"2024-09-21T01:03:08.296157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485309+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:02:10.485309+0000","last_peered":"2024-09-21T01:02:10.485309+0000","last_clean":"2024-09-21T01:02:10.485309+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:02:10.485309+0000","last_undegraded":"2024-09-21T01:02:10.485309+0000","last_fullsized":"2024-09-21T01:02:10.485309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.144022+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:02:11.144022+0000","last_peered":"2024-09-21T01:02:11.144022+0000","last_clean":"2024-09-21T01:02:11.144022+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:02:11.144022+0000","last_undegraded":"2024-09-21T01:02:11.144022+0000","last_fullsized":"2024-09-21T01:02:11.144022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.308007+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:03:08.308007+0000","last_peered":"2024-09-21T01:03:08.308007+0000","last_clean":"2024-09-21T01:03:08.308007+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:03:08.308007+0000","last_undegraded":"2024-09-21T01:03:08.308007+0000","last_fullsized":"2024-09-21T01:03:08.308007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487364+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:02:10.487364+0000","last_peered":"2024-09-21T01:02:10.487364+0000","last_clean":"2024-09-21T01:02:10.487364+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:02:10.487364+0000","last_undegraded":"2024-09-21T01:02:10.487364+0000","last_fullsized":"2024-09-21T01:02:10.487364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137270+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:02:11.137270+0000","last_peered":"2024-09-21T01:02:11.137270+0000","last_clean":"2024-09-21T01:02:11.137270+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:02:11.137270+0000","last_undegraded":"2024-09-21T01:02:11.137270+0000","last_fullsized":"2024-09-21T01:02:11.137270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307119+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:03:08.307119+0000","last_peered":"2024-09-21T01:03:08.307119+0000","last_clean":"2024-09-21T01:03:08.307119+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:03:08.307119+0000","last_undegraded":"2024-09-21T01:03:08.307119+0000","last_fullsized":"2024-09-21T01:03:08.307119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487404+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:02:10.487404+0000","last_peered":"2024-09-21T01:02:10.487404+0000","last_clean":"2024-09-21T01:02:10.487404+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:02:10.487404+0000","last_undegraded":"2024-09-21T01:02:10.487404+0000","last_fullsized":"2024-09-21T01:02:10.487404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137027+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:02:11.137027+0000","last_peered":"2024-09-21T01:02:11.137027+0000","last_clean":"2024-09-21T01:02:11.137027+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:02:11.137027+0000","last_undegraded":"2024-09-21T01:02:11.137027+0000","last_fullsized":"2024-09-21T01:02:11.137027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301310+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:03:08.301310+0000","last_peered":"2024-09-21T01:03:08.301310+0000","last_clean":"2024-09-21T01:03:08.301310+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:03:08.301310+0000","last_undegraded":"2024-09-21T01:03:08.301310+0000","last_fullsized":"2024-09-21T01:03:08.301310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487356+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:02:10.487356+0000","last_peered":"2024-09-21T01:02:10.487356+0000","last_clean":"2024-09-21T01:02:10.487356+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:02:10.487356+0000","last_undegraded":"2024-09-21T01:02:10.487356+0000","last_fullsized":"2024-09-21T01:02:10.487356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.306792+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:03:08.306792+0000","last_peered":"2024-09-21T01:03:08.306792+0000","last_clean":"2024-09-21T01:03:08.306792+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:03:08.306792+0000","last_undegraded":"2024-09-21T01:03:08.306792+0000","last_fullsized":"2024-09-21T01:03:08.306792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136891+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:02:11.136891+0000","last_peered":"2024-09-21T01:02:11.136891+0000","last_clean":"2024-09-21T01:02:11.136891+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:02:11.136891+0000","last_undegraded":"2024-09-21T01:02:11.136891+0000","last_fullsized":"2024-09-21T01:02:11.136891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488041+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:02:10.488041+0000","last_peered":"2024-09-21T01:02:10.488041+0000","last_clean":"2024-09-21T01:02:10.488041+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:02:10.488041+0000","last_undegraded":"2024-09-21T01:02:10.488041+0000","last_fullsized":"2024-09-21T01:02:10.488041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304480+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:03:08.304480+0000","last_peered":"2024-09-21T01:03:08.304480+0000","last_clean":"2024-09-21T01:03:08.304480+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:03:08.304480+0000","last_undegraded":"2024-09-21T01:03:08.304480+0000","last_fullsized":"2024-09-21T01:03:08.304480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485418+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:02:10.485418+0000","last_peered":"2024-09-21T01:02:10.485418+0000","last_clean":"2024-09-21T01:02:10.485418+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:02:10.485418+0000","last_undegraded":"2024-09-21T01:02:10.485418+0000","last_fullsized":"2024-09-21T01:02:10.485418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487939+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:02:10.487939+0000","last_peered":"2024-09-21T01:02:10.487939+0000","last_clean":"2024-09-21T01:02:10.487939+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:02:10.487939+0000","last_undegraded":"2024-09-21T01:02:10.487939+0000","last_fullsized":"2024-09-21T01:02:10.487939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304538+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:03:08.304538+0000","last_peered":"2024-09-21T01:03:08.304538+0000","last_clean":"2024-09-21T01:03:08.304538+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:03:08.304538+0000","last_undegraded":"2024-09-21T01:03:08.304538+0000","last_fullsized":"2024-09-21T01:03:08.304538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143393+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:02:11.143393+0000","last_peered":"2024-09-21T01:02:11.143393+0000","last_clean":"2024-09-21T01:02:11.143393+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:02:11.143393+0000","last_undegraded":"2024-09-21T01:02:11.143393+0000","last_fullsized":"2024-09-21T01:02:11.143393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488402+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:02:10.488402+0000","last_peered":"2024-09-21T01:02:10.488402+0000","last_clean":"2024-09-21T01:02:10.488402+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:02:10.488402+0000","last_undegraded":"2024-09-21T01:02:10.488402+0000","last_fullsized":"2024-09-21T01:02:10.488402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.254592+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:03:08.254592+0000","last_peered":"2024-09-21T01:03:08.254592+0000","last_clean":"2024-09-21T01:03:08.254592+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:03:08.254592+0000","last_undegraded":"2024-09-21T01:03:08.254592+0000","last_fullsized":"2024-09-21T01:03:08.254592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964583+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:02:10.964583+0000","last_peered":"2024-09-21T01:02:10.964583+0000","last_clean":"2024-09-21T01:02:10.964583+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:02:10.964583+0000","last_undegraded":"2024-09-21T01:02:10.964583+0000","last_fullsized":"2024-09-21T01:02:10.964583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488121+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:02:10.488121+0000","last_peered":"2024-09-21T01:02:10.488121+0000","last_clean":"2024-09-21T01:02:10.488121+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:02:10.488121+0000","last_undegraded":"2024-09-21T01:02:10.488121+0000","last_fullsized":"2024-09-21T01:02:10.488121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293749+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:03:08.293749+0000","last_peered":"2024-09-21T01:03:08.293749+0000","last_clean":"2024-09-21T01:03:08.293749+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:03:08.293749+0000","last_undegraded":"2024-09-21T01:03:08.293749+0000","last_fullsized":"2024-09-21T01:03:08.293749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488427+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:02:10.488427+0000","last_peered":"2024-09-21T01:02:10.488427+0000","last_clean":"2024-09-21T01:02:10.488427+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:02:10.488427+0000","last_undegraded":"2024-09-21T01:02:10.488427+0000","last_fullsized":"2024-09-21T01:02:10.488427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485629+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:02:10.485629+0000","last_peered":"2024-09-21T01:02:10.485629+0000","last_clean":"2024-09-21T01:02:10.485629+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:02:10.485629+0000","last_undegraded":"2024-09-21T01:02:10.485629+0000","last_fullsized":"2024-09-21T01:02:10.485629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301452+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:03:08.301452+0000","last_peered":"2024-09-21T01:03:08.301452+0000","last_clean":"2024-09-21T01:03:08.301452+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:03:08.301452+0000","last_undegraded":"2024-09-21T01:03:08.301452+0000","last_fullsized":"2024-09-21T01:03:08.301452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'97","reported_seq":231,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:47.291053+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:03:47.291053+0000","last_peered":"2024-09-21T01:03:47.291053+0000","last_clean":"2024-09-21T01:03:47.291053+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:03:47.291053+0000","last_undegraded":"2024-09-21T01:03:47.291053+0000","last_fullsized":"2024-09-21T01:03:47.291053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":178,"num_write_kb":1221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142973+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:02:11.142973+0000","last_peered":"2024-09-21T01:02:11.142973+0000","last_clean":"2024-09-21T01:02:11.142973+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:02:11.142973+0000","last_undegraded":"2024-09-21T01:02:11.142973+0000","last_fullsized":"2024-09-21T01:02:11.142973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296507+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:03:08.296507+0000","last_peered":"2024-09-21T01:03:08.296507+0000","last_clean":"2024-09-21T01:03:08.296507+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:03:08.296507+0000","last_undegraded":"2024-09-21T01:03:08.296507+0000","last_fullsized":"2024-09-21T01:03:08.296507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137262+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:02:11.137262+0000","last_peered":"2024-09-21T01:02:11.137262+0000","last_clean":"2024-09-21T01:02:11.137262+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:02:11.137262+0000","last_undegraded":"2024-09-21T01:02:11.137262+0000","last_fullsized":"2024-09-21T01:02:11.137262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316968+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:02:11.316968+0000","last_peered":"2024-09-21T01:02:11.316968+0000","last_clean":"2024-09-21T01:02:11.316968+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:02:11.316968+0000","last_undegraded":"2024-09-21T01:02:11.316968+0000","last_fullsized":"2024-09-21T01:02:11.316968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.287755+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:03:08.287755+0000","last_peered":"2024-09-21T01:03:08.287755+0000","last_clean":"2024-09-21T01:03:08.287755+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:03:08.287755+0000","last_undegraded":"2024-09-21T01:03:08.287755+0000","last_fullsized":"2024-09-21T01:03:08.287755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488662+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:02:10.488662+0000","last_peered":"2024-09-21T01:02:10.488662+0000","last_clean":"2024-09-21T01:02:10.488662+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:02:10.488662+0000","last_undegraded":"2024-09-21T01:02:10.488662+0000","last_fullsized":"2024-09-21T01:02:10.488662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136558+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:02:11.136558+0000","last_peered":"2024-09-21T01:02:11.136558+0000","last_clean":"2024-09-21T01:02:11.136558+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:02:11.136558+0000","last_undegraded":"2024-09-21T01:02:11.136558+0000","last_fullsized":"2024-09-21T01:02:11.136558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293464+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:03:08.293464+0000","last_peered":"2024-09-21T01:03:08.293464+0000","last_clean":"2024-09-21T01:03:08.293464+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:03:08.293464+0000","last_undegraded":"2024-09-21T01:03:08.293464+0000","last_fullsized":"2024-09-21T01:03:08.293464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490887+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:02:10.490887+0000","last_peered":"2024-09-21T01:02:10.490887+0000","last_clean":"2024-09-21T01:02:10.490887+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:02:10.490887+0000","last_undegraded":"2024-09-21T01:02:10.490887+0000","last_fullsized":"2024-09-21T01:02:10.490887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484822+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:02:10.484822+0000","last_peered":"2024-09-21T01:02:10.484822+0000","last_clean":"2024-09-21T01:02:10.484822+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:02:10.484822+0000","last_undegraded":"2024-09-21T01:02:10.484822+0000","last_fullsized":"2024-09-21T01:02:10.484822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.292886+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:03:08.292886+0000","last_peered":"2024-09-21T01:03:08.292886+0000","last_clean":"2024-09-21T01:03:08.292886+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:03:08.292886+0000","last_undegraded":"2024-09-21T01:03:08.292886+0000","last_fullsized":"2024-09-21T01:03:08.292886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:45.603143+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:03:45.603143+0000","last_peered":"2024-09-21T01:03:45.603143+0000","last_clean":"2024-09-21T01:03:45.603143+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:03:45.603143+0000","last_undegraded":"2024-09-21T01:03:45.603143+0000","last_fullsized":"2024-09-21T01:03:45.603143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485099+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:02:10.485099+0000","last_peered":"2024-09-21T01:02:10.485099+0000","last_clean":"2024-09-21T01:02:10.485099+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:02:10.485099+0000","last_undegraded":"2024-09-21T01:02:10.485099+0000","last_fullsized":"2024-09-21T01:02:10.485099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296516+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:03:08.296516+0000","last_peered":"2024-09-21T01:03:08.296516+0000","last_clean":"2024-09-21T01:03:08.296516+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:03:08.296516+0000","last_undegraded":"2024-09-21T01:03:08.296516+0000","last_fullsized":"2024-09-21T01:03:08.296516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490838+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:02:10.490838+0000","last_peered":"2024-09-21T01:02:10.490838+0000","last_clean":"2024-09-21T01:02:10.490838+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:02:10.490838+0000","last_undegraded":"2024-09-21T01:02:10.490838+0000","last_fullsized":"2024-09-21T01:02:10.490838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489999+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:02:10.489999+0000","last_peered":"2024-09-21T01:02:10.489999+0000","last_clean":"2024-09-21T01:02:10.489999+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:02:10.489999+0000","last_undegraded":"2024-09-21T01:02:10.489999+0000","last_fullsized":"2024-09-21T01:02:10.489999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301050+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:03:08.301050+0000","last_peered":"2024-09-21T01:03:08.301050+0000","last_clean":"2024-09-21T01:03:08.301050+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:03:08.301050+0000","last_undegraded":"2024-09-21T01:03:08.301050+0000","last_fullsized":"2024-09-21T01:03:08.301050+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487931+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:02:10.487931+0000","last_peered":"2024-09-21T01:02:10.487931+0000","last_clean":"2024-09-21T01:02:10.487931+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:02:10.487931+0000","last_undegraded":"2024-09-21T01:02:10.487931+0000","last_fullsized":"2024-09-21T01:02:10.487931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490885+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:02:10.490885+0000","last_peered":"2024-09-21T01:02:10.490885+0000","last_clean":"2024-09-21T01:02:10.490885+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:02:10.490885+0000","last_undegraded":"2024-09-21T01:02:10.490885+0000","last_fullsized":"2024-09-21T01:02:10.490885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490252+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:02:10.490252+0000","last_peered":"2024-09-21T01:02:10.490252+0000","last_clean":"2024-09-21T01:02:10.490252+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:02:10.490252+0000","last_undegraded":"2024-09-21T01:02:10.490252+0000","last_fullsized":"2024-09-21T01:02:10.490252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301604+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:03:08.301604+0000","last_peered":"2024-09-21T01:03:08.301604+0000","last_clean":"2024-09-21T01:03:08.301604+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:03:08.301604+0000","last_undegraded":"2024-09-21T01:03:08.301604+0000","last_fullsized":"2024-09-21T01:03:08.301604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485952+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:02:10.485952+0000","last_peered":"2024-09-21T01:02:10.485952+0000","last_clean":"2024-09-21T01:02:10.485952+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:02:10.485952+0000","last_undegraded":"2024-09-21T01:02:10.485952+0000","last_fullsized":"2024-09-21T01:02:10.485952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.965031+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:02:10.965031+0000","last_peered":"2024-09-21T01:02:10.965031+0000","last_clean":"2024-09-21T01:02:10.965031+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:02:10.965031+0000","last_undegraded":"2024-09-21T01:02:10.965031+0000","last_fullsized":"2024-09-21T01:02:10.965031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300921+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:03:08.300921+0000","last_peered":"2024-09-21T01:03:08.300921+0000","last_clean":"2024-09-21T01:03:08.300921+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:03:08.300921+0000","last_undegraded":"2024-09-21T01:03:08.300921+0000","last_fullsized":"2024-09-21T01:03:08.300921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'60","reported_seq":133,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:46.298866+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:03:46.298866+0000","last_peered":"2024-09-21T01:03:46.298866+0000","last_clean":"2024-09-21T01:03:46.298866+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:03:46.298866+0000","last_undegraded":"2024-09-21T01:03:46.298866+0000","last_fullsized":"2024-09-21T01:03:46.298866+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645580+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:02:10.645580+0000","last_peered":"2024-09-21T01:02:10.645580+0000","last_clean":"2024-09-21T01:02:10.645580+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:02:10.645580+0000","last_undegraded":"2024-09-21T01:02:10.645580+0000","last_fullsized":"2024-09-21T01:02:10.645580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302551+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:03:08.302551+0000","last_peered":"2024-09-21T01:03:08.302551+0000","last_clean":"2024-09-21T01:03:08.302551+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:03:08.302551+0000","last_undegraded":"2024-09-21T01:03:08.302551+0000","last_fullsized":"2024-09-21T01:03:08.302551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485106+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:02:10.485106+0000","last_peered":"2024-09-21T01:02:10.485106+0000","last_clean":"2024-09-21T01:02:10.485106+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:02:10.485106+0000","last_undegraded":"2024-09-21T01:02:10.485106+0000","last_fullsized":"2024-09-21T01:02:10.485106+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489060+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:02:10.489060+0000","last_peered":"2024-09-21T01:02:10.489060+0000","last_clean":"2024-09-21T01:02:10.489060+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:02:10.489060+0000","last_undegraded":"2024-09-21T01:02:10.489060+0000","last_fullsized":"2024-09-21T01:02:10.489060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290221+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:03:08.290221+0000","last_peered":"2024-09-21T01:03:08.290221+0000","last_clean":"2024-09-21T01:03:08.290221+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:03:08.290221+0000","last_undegraded":"2024-09-21T01:03:08.290221+0000","last_fullsized":"2024-09-21T01:03:08.290221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:25.005750+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:03:25.005750+0000","last_peered":"2024-09-21T01:03:25.005750+0000","last_clean":"2024-09-21T01:03:25.005750+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:03:25.005750+0000","last_undegraded":"2024-09-21T01:03:25.005750+0000","last_fullsized":"2024-09-21T01:03:25.005750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489786+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:02:10.489786+0000","last_peered":"2024-09-21T01:02:10.489786+0000","last_clean":"2024-09-21T01:02:10.489786+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:02:10.489786+0000","last_undegraded":"2024-09-21T01:02:10.489786+0000","last_fullsized":"2024-09-21T01:02:10.489786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295651+0000","last_change":"2024-09-21T01:02:10.966004+0000","last_active":"2024-09-21T01:03:08.295651+0000","last_peered":"2024-09-21T01:03:08.295651+0000","last_clean":"2024-09-21T01:03:08.295651+0000","last_became_active":"2024-09-21T01:02:10.965645+0000","last_became_peered":"2024-09-21T01:02:10.965645+0000","last_unstale":"2024-09-21T01:03:08.295651+0000","last_undegraded":"2024-09-21T01:03:08.295651+0000","last_fullsized":"2024-09-21T01:03:08.295651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:12.070017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136620+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:02:11.136620+0000","last_peered":"2024-09-21T01:02:11.136620+0000","last_clean":"2024-09-21T01:02:11.136620+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:02:11.136620+0000","last_undegraded":"2024-09-21T01:02:11.136620+0000","last_fullsized":"2024-09-21T01:02:11.136620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485028+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:02:10.485028+0000","last_peered":"2024-09-21T01:02:10.485028+0000","last_clean":"2024-09-21T01:02:10.485028+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:02:10.485028+0000","last_undegraded":"2024-09-21T01:02:10.485028+0000","last_fullsized":"2024-09-21T01:02:10.485028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300660+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:03:08.300660+0000","last_peered":"2024-09-21T01:03:08.300660+0000","last_clean":"2024-09-21T01:03:08.300660+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:03:08.300660+0000","last_undegraded":"2024-09-21T01:03:08.300660+0000","last_fullsized":"2024-09-21T01:03:08.300660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:44.602219+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:03:44.602219+0000","last_peered":"2024-09-21T01:03:44.602219+0000","last_clean":"2024-09-21T01:03:44.602219+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:03:44.602219+0000","last_undegraded":"2024-09-21T01:03:44.602219+0000","last_fullsized":"2024-09-21T01:03:44.602219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1431,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489132+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:02:10.489132+0000","last_peered":"2024-09-21T01:02:10.489132+0000","last_clean":"2024-09-21T01:02:10.489132+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:02:10.489132+0000","last_undegraded":"2024-09-21T01:02:10.489132+0000","last_fullsized":"2024-09-21T01:02:10.489132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304638+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:03:08.304638+0000","last_peered":"2024-09-21T01:03:08.304638+0000","last_clean":"2024-09-21T01:03:08.304638+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:03:08.304638+0000","last_undegraded":"2024-09-21T01:03:08.304638+0000","last_fullsized":"2024-09-21T01:03:08.304638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489455+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:02:10.489455+0000","last_peered":"2024-09-21T01:02:10.489455+0000","last_clean":"2024-09-21T01:02:10.489455+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:02:10.489455+0000","last_undegraded":"2024-09-21T01:02:10.489455+0000","last_fullsized":"2024-09-21T01:02:10.489455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316922+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:02:11.316922+0000","last_peered":"2024-09-21T01:02:11.316922+0000","last_clean":"2024-09-21T01:02:11.316922+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:02:11.316922+0000","last_undegraded":"2024-09-21T01:02:11.316922+0000","last_fullsized":"2024-09-21T01:02:11.316922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298020+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:03:08.298020+0000","last_peered":"2024-09-21T01:03:08.298020+0000","last_clean":"2024-09-21T01:03:08.298020+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:03:08.298020+0000","last_undegraded":"2024-09-21T01:03:08.298020+0000","last_fullsized":"2024-09-21T01:03:08.298020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485567+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:02:10.485567+0000","last_peered":"2024-09-21T01:02:10.485567+0000","last_clean":"2024-09-21T01:02:10.485567+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:02:10.485567+0000","last_undegraded":"2024-09-21T01:02:10.485567+0000","last_fullsized":"2024-09-21T01:02:10.485567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489683+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:02:10.489683+0000","last_peered":"2024-09-21T01:02:10.489683+0000","last_clean":"2024-09-21T01:02:10.489683+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:02:10.489683+0000","last_undegraded":"2024-09-21T01:02:10.489683+0000","last_fullsized":"2024-09-21T01:02:10.489683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297147+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:03:08.297147+0000","last_peered":"2024-09-21T01:03:08.297147+0000","last_clean":"2024-09-21T01:03:08.297147+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:03:08.297147+0000","last_undegraded":"2024-09-21T01:03:08.297147+0000","last_fullsized":"2024-09-21T01:03:08.297147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:44.576396+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:03:44.576396+0000","last_peered":"2024-09-21T01:03:44.576396+0000","last_clean":"2024-09-21T01:03:44.576396+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:03:44.576396+0000","last_undegraded":"2024-09-21T01:03:44.576396+0000","last_fullsized":"2024-09-21T01:03:44.576396+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73667,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":203,"num_write":42,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488195+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:02:10.488195+0000","last_peered":"2024-09-21T01:02:10.488195+0000","last_clean":"2024-09-21T01:02:10.488195+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:02:10.488195+0000","last_undegraded":"2024-09-21T01:02:10.488195+0000","last_fullsized":"2024-09-21T01:02:10.488195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.305890+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:03:08.305890+0000","last_peered":"2024-09-21T01:03:08.305890+0000","last_clean":"2024-09-21T01:03:08.305890+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:03:08.305890+0000","last_undegraded":"2024-09-21T01:03:08.305890+0000","last_fullsized":"2024-09-21T01:03:08.305890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489820+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:02:10.489820+0000","last_peered":"2024-09-21T01:02:10.489820+0000","last_clean":"2024-09-21T01:02:10.489820+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:02:10.489820+0000","last_undegraded":"2024-09-21T01:02:10.489820+0000","last_fullsized":"2024-09-21T01:02:10.489820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487710+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:02:10.487710+0000","last_peered":"2024-09-21T01:02:10.487710+0000","last_clean":"2024-09-21T01:02:10.487710+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:02:10.487710+0000","last_undegraded":"2024-09-21T01:02:10.487710+0000","last_fullsized":"2024-09-21T01:02:10.487710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297611+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:03:08.297611+0000","last_peered":"2024-09-21T01:03:08.297611+0000","last_clean":"2024-09-21T01:03:08.297611+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:03:08.297611+0000","last_undegraded":"2024-09-21T01:03:08.297611+0000","last_fullsized":"2024-09-21T01:03:08.297611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490763+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:02:10.490763+0000","last_peered":"2024-09-21T01:02:10.490763+0000","last_clean":"2024-09-21T01:02:10.490763+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:02:10.490763+0000","last_undegraded":"2024-09-21T01:02:10.490763+0000","last_fullsized":"2024-09-21T01:02:10.490763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487966+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:02:10.487966+0000","last_peered":"2024-09-21T01:02:10.487966+0000","last_clean":"2024-09-21T01:02:10.487966+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:02:10.487966+0000","last_undegraded":"2024-09-21T01:02:10.487966+0000","last_fullsized":"2024-09-21T01:02:10.487966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.305939+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:03:08.305939+0000","last_peered":"2024-09-21T01:03:08.305939+0000","last_clean":"2024-09-21T01:03:08.305939+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:03:08.305939+0000","last_undegraded":"2024-09-21T01:03:08.305939+0000","last_fullsized":"2024-09-21T01:03:08.305939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:43.573250+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:03:43.573250+0000","last_peered":"2024-09-21T01:03:43.573250+0000","last_clean":"2024-09-21T01:03:43.573250+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:03:43.573250+0000","last_undegraded":"2024-09-21T01:03:43.573250+0000","last_fullsized":"2024-09-21T01:03:43.573250+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143140+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:02:11.143140+0000","last_peered":"2024-09-21T01:02:11.143140+0000","last_clean":"2024-09-21T01:02:11.143140+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:02:11.143140+0000","last_undegraded":"2024-09-21T01:02:11.143140+0000","last_fullsized":"2024-09-21T01:02:11.143140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645756+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:02:10.645756+0000","last_peered":"2024-09-21T01:02:10.645756+0000","last_clean":"2024-09-21T01:02:10.645756+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:02:10.645756+0000","last_undegraded":"2024-09-21T01:02:10.645756+0000","last_fullsized":"2024-09-21T01:02:10.645756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301916+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:03:08.301916+0000","last_peered":"2024-09-21T01:03:08.301916+0000","last_clean":"2024-09-21T01:03:08.301916+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:03:08.301916+0000","last_undegraded":"2024-09-21T01:03:08.301916+0000","last_fullsized":"2024-09-21T01:03:08.301916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489429+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:02:10.489429+0000","last_peered":"2024-09-21T01:02:10.489429+0000","last_clean":"2024-09-21T01:02:10.489429+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:02:10.489429+0000","last_undegraded":"2024-09-21T01:02:10.489429+0000","last_fullsized":"2024-09-21T01:02:10.489429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317166+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:02:11.317166+0000","last_peered":"2024-09-21T01:02:11.317166+0000","last_clean":"2024-09-21T01:02:11.317166+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:02:11.317166+0000","last_undegraded":"2024-09-21T01:02:11.317166+0000","last_fullsized":"2024-09-21T01:02:11.317166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297901+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:03:08.297901+0000","last_peered":"2024-09-21T01:03:08.297901+0000","last_clean":"2024-09-21T01:03:08.297901+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:03:08.297901+0000","last_undegraded":"2024-09-21T01:03:08.297901+0000","last_fullsized":"2024-09-21T01:03:08.297901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484277+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:02:10.484277+0000","last_peered":"2024-09-21T01:02:10.484277+0000","last_clean":"2024-09-21T01:02:10.484277+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:02:10.484277+0000","last_undegraded":"2024-09-21T01:02:10.484277+0000","last_fullsized":"2024-09-21T01:02:10.484277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517818+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:02:10.517818+0000","last_peered":"2024-09-21T01:02:10.517818+0000","last_clean":"2024-09-21T01:02:10.517818+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:02:10.517818+0000","last_undegraded":"2024-09-21T01:02:10.517818+0000","last_fullsized":"2024-09-21T01:02:10.517818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302502+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:03:08.302502+0000","last_peered":"2024-09-21T01:03:08.302502+0000","last_clean":"2024-09-21T01:03:08.302502+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:03:08.302502+0000","last_undegraded":"2024-09-21T01:03:08.302502+0000","last_fullsized":"2024-09-21T01:03:08.302502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485639+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:02:10.485639+0000","last_peered":"2024-09-21T01:02:10.485639+0000","last_clean":"2024-09-21T01:02:10.485639+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:02:10.485639+0000","last_undegraded":"2024-09-21T01:02:10.485639+0000","last_fullsized":"2024-09-21T01:02:10.485639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:45.577011+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:03:45.577011+0000","last_peered":"2024-09-21T01:03:45.577011+0000","last_clean":"2024-09-21T01:03:45.577011+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:03:45.577011+0000","last_undegraded":"2024-09-21T01:03:45.577011+0000","last_fullsized":"2024-09-21T01:03:45.577011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295018+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:03:08.295018+0000","last_peered":"2024-09-21T01:03:08.295018+0000","last_clean":"2024-09-21T01:03:08.295018+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:03:08.295018+0000","last_undegraded":"2024-09-21T01:03:08.295018+0000","last_fullsized":"2024-09-21T01:03:08.295018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485589+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:02:10.485589+0000","last_peered":"2024-09-21T01:02:10.485589+0000","last_clean":"2024-09-21T01:02:10.485589+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:02:10.485589+0000","last_undegraded":"2024-09-21T01:02:10.485589+0000","last_fullsized":"2024-09-21T01:02:10.485589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.294307+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:03:08.294307+0000","last_peered":"2024-09-21T01:03:08.294307+0000","last_clean":"2024-09-21T01:03:08.294307+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:03:08.294307+0000","last_undegraded":"2024-09-21T01:03:08.294307+0000","last_fullsized":"2024-09-21T01:03:08.294307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297962+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:03:08.297962+0000","last_peered":"2024-09-21T01:03:08.297962+0000","last_clean":"2024-09-21T01:03:08.297962+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:03:08.297962+0000","last_undegraded":"2024-09-21T01:03:08.297962+0000","last_fullsized":"2024-09-21T01:03:08.297962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.276740+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:03:08.276740+0000","last_peered":"2024-09-21T01:03:08.276740+0000","last_clean":"2024-09-21T01:03:08.276740+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:03:08.276740+0000","last_undegraded":"2024-09-21T01:03:08.276740+0000","last_fullsized":"2024-09-21T01:03:08.276740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.299954+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:03:08.299954+0000","last_peered":"2024-09-21T01:03:08.299954+0000","last_clean":"2024-09-21T01:03:08.299954+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:03:08.299954+0000","last_undegraded":"2024-09-21T01:03:08.299954+0000","last_fullsized":"2024-09-21T01:03:08.299954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.303193+0000","last_change":"2024-09-21T01:02:10.647066+0000","last_active":"2024-09-21T01:03:08.303193+0000","last_peered":"2024-09-21T01:03:08.303193+0000","last_clean":"2024-09-21T01:03:08.303193+0000","last_became_active":"2024-09-21T01:02:10.646487+0000","last_became_peered":"2024-09-21T01:02:10.646487+0000","last_unstale":"2024-09-21T01:03:08.303193+0000","last_undegraded":"2024-09-21T01:03:08.303193+0000","last_fullsized":"2024-09-21T01:03:08.303193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:19:28.445094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296459+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:03:08.296459+0000","last_peered":"2024-09-21T01:03:08.296459+0000","last_clean":"2024-09-21T01:03:08.296459+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:03:08.296459+0000","last_undegraded":"2024-09-21T01:03:08.296459+0000","last_fullsized":"2024-09-21T01:03:08.296459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295660+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:03:08.295660+0000","last_peered":"2024-09-21T01:03:08.295660+0000","last_clean":"2024-09-21T01:03:08.295660+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:03:08.295660+0000","last_undegraded":"2024-09-21T01:03:08.295660+0000","last_fullsized":"2024-09-21T01:03:08.295660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.291803+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:03:08.291803+0000","last_peered":"2024-09-21T01:03:08.291803+0000","last_clean":"2024-09-21T01:03:08.291803+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:03:08.291803+0000","last_undegraded":"2024-09-21T01:03:08.291803+0000","last_fullsized":"2024-09-21T01:03:08.291803+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295455+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:03:08.295455+0000","last_peered":"2024-09-21T01:03:08.295455+0000","last_clean":"2024-09-21T01:03:08.295455+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:03:08.295455+0000","last_undegraded":"2024-09-21T01:03:08.295455+0000","last_fullsized":"2024-09-21T01:03:08.295455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293894+0000","last_change":"2024-09-21T01:02:11.318295+0000","last_active":"2024-09-21T01:03:08.293894+0000","last_peered":"2024-09-21T01:03:08.293894+0000","last_clean":"2024-09-21T01:03:08.293894+0000","last_became_active":"2024-09-21T01:02:11.317912+0000","last_became_peered":"2024-09-21T01:02:11.317912+0000","last_unstale":"2024-09-21T01:03:08.293894+0000","last_undegraded":"2024-09-21T01:03:08.293894+0000","last_fullsized":"2024-09-21T01:03:08.293894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:53:47.063340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307810+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:03:08.307810+0000","last_peered":"2024-09-21T01:03:08.307810+0000","last_clean":"2024-09-21T01:03:08.307810+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:03:08.307810+0000","last_undegraded":"2024-09-21T01:03:08.307810+0000","last_fullsized":"2024-09-21T01:03:08.307810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298761+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:03:08.298761+0000","last_peered":"2024-09-21T01:03:08.298761+0000","last_clean":"2024-09-21T01:03:08.298761+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:03:08.298761+0000","last_undegraded":"2024-09-21T01:03:08.298761+0000","last_fullsized":"2024-09-21T01:03:08.298761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296634+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:03:08.296634+0000","last_peered":"2024-09-21T01:03:08.296634+0000","last_clean":"2024-09-21T01:03:08.296634+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:03:08.296634+0000","last_undegraded":"2024-09-21T01:03:08.296634+0000","last_fullsized":"2024-09-21T01:03:08.296634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290097+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:03:08.290097+0000","last_peered":"2024-09-21T01:03:08.290097+0000","last_clean":"2024-09-21T01:03:08.290097+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:03:08.290097+0000","last_undegraded":"2024-09-21T01:03:08.290097+0000","last_fullsized":"2024-09-21T01:03:08.290097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.299853+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:03:08.299853+0000","last_peered":"2024-09-21T01:03:08.299853+0000","last_clean":"2024-09-21T01:03:08.299853+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:03:08.299853+0000","last_undegraded":"2024-09-21T01:03:08.299853+0000","last_fullsized":"2024-09-21T01:03:08.299853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307223+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:03:08.307223+0000","last_peered":"2024-09-21T01:03:08.307223+0000","last_clean":"2024-09-21T01:03:08.307223+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:03:08.307223+0000","last_undegraded":"2024-09-21T01:03:08.307223+0000","last_fullsized":"2024-09-21T01:03:08.307223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307652+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:03:08.307652+0000","last_peered":"2024-09-21T01:03:08.307652+0000","last_clean":"2024-09-21T01:03:08.307652+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:03:08.307652+0000","last_undegraded":"2024-09-21T01:03:08.307652+0000","last_fullsized":"2024-09-21T01:03:08.307652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298544+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:03:08.298544+0000","last_peered":"2024-09-21T01:03:08.298544+0000","last_clean":"2024-09-21T01:03:08.298544+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:03:08.298544+0000","last_undegraded":"2024-09-21T01:03:08.298544+0000","last_fullsized":"2024-09-21T01:03:08.298544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304775+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:03:08.304775+0000","last_peered":"2024-09-21T01:03:08.304775+0000","last_clean":"2024-09-21T01:03:08.304775+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:03:08.304775+0000","last_undegraded":"2024-09-21T01:03:08.304775+0000","last_fullsized":"2024-09-21T01:03:08.304775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300375+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:03:08.300375+0000","last_peered":"2024-09-21T01:03:08.300375+0000","last_clean":"2024-09-21T01:03:08.300375+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:03:08.300375+0000","last_undegraded":"2024-09-21T01:03:08.300375+0000","last_fullsized":"2024-09-21T01:03:08.300375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297218+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:03:08.297218+0000","last_peered":"2024-09-21T01:03:08.297218+0000","last_clean":"2024-09-21T01:03:08.297218+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:03:08.297218+0000","last_undegraded":"2024-09-21T01:03:08.297218+0000","last_fullsized":"2024-09-21T01:03:08.297218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302497+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:03:08.302497+0000","last_peered":"2024-09-21T01:03:08.302497+0000","last_clean":"2024-09-21T01:03:08.302497+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:03:08.302497+0000","last_undegraded":"2024-09-21T01:03:08.302497+0000","last_fullsized":"2024-09-21T01:03:08.302497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290792+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:03:08.290792+0000","last_peered":"2024-09-21T01:03:08.290792+0000","last_clean":"2024-09-21T01:03:08.290792+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:03:08.290792+0000","last_undegraded":"2024-09-21T01:03:08.290792+0000","last_fullsized":"2024-09-21T01:03:08.290792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.306589+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:03:08.306589+0000","last_peered":"2024-09-21T01:03:08.306589+0000","last_clean":"2024-09-21T01:03:08.306589+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:03:08.306589+0000","last_undegraded":"2024-09-21T01:03:08.306589+0000","last_fullsized":"2024-09-21T01:03:08.306589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301598+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:03:08.301598+0000","last_peered":"2024-09-21T01:03:08.301598+0000","last_clean":"2024-09-21T01:03:08.301598+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:03:08.301598+0000","last_undegraded":"2024-09-21T01:03:08.301598+0000","last_fullsized":"2024-09-21T01:03:08.301598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.294226+0000","last_change":"2024-09-21T01:02:11.138284+0000","last_active":"2024-09-21T01:03:08.294226+0000","last_peered":"2024-09-21T01:03:08.294226+0000","last_clean":"2024-09-21T01:03:08.294226+0000","last_became_active":"2024-09-21T01:02:11.137782+0000","last_became_peered":"2024-09-21T01:02:11.137782+0000","last_unstale":"2024-09-21T01:03:08.294226+0000","last_undegraded":"2024-09-21T01:03:08.294226+0000","last_fullsized":"2024-09-21T01:03:08.294226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:57.753015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298298+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:03:08.298298+0000","last_peered":"2024-09-21T01:03:08.298298+0000","last_clean":"2024-09-21T01:03:08.298298+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:03:08.298298+0000","last_undegraded":"2024-09-21T01:03:08.298298+0000","last_fullsized":"2024-09-21T01:03:08.298298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.292151+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:03:08.292151+0000","last_peered":"2024-09-21T01:03:08.292151+0000","last_clean":"2024-09-21T01:03:08.292151+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:03:08.292151+0000","last_undegraded":"2024-09-21T01:03:08.292151+0000","last_fullsized":"2024-09-21T01:03:08.292151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297492+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:03:08.297492+0000","last_peered":"2024-09-21T01:03:08.297492+0000","last_clean":"2024-09-21T01:03:08.297492+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:03:08.297492+0000","last_undegraded":"2024-09-21T01:03:08.297492+0000","last_fullsized":"2024-09-21T01:03:08.297492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307974+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:03:08.307974+0000","last_peered":"2024-09-21T01:03:08.307974+0000","last_clean":"2024-09-21T01:03:08.307974+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:03:08.307974+0000","last_undegraded":"2024-09-21T01:03:08.307974+0000","last_fullsized":"2024-09-21T01:03:08.307974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.303566+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:03:08.303566+0000","last_peered":"2024-09-21T01:03:08.303566+0000","last_clean":"2024-09-21T01:03:08.303566+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:03:08.303566+0000","last_undegraded":"2024-09-21T01:03:08.303566+0000","last_fullsized":"2024-09-21T01:03:08.303566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307001+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:03:08.307001+0000","last_peered":"2024-09-21T01:03:08.307001+0000","last_clean":"2024-09-21T01:03:08.307001+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:03:08.307001+0000","last_undegraded":"2024-09-21T01:03:08.307001+0000","last_fullsized":"2024-09-21T01:03:08.307001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":865644,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1117,"num_read_kb":773,"num_write":412,"num_write_kb":2231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1800931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678067,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":535286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874295,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28724,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716428,"statfs":{"total":95995035648,"available":95965622272,"internally_reserved":0,"allocated":1740800,"data_stored":1196789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037823,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28444,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716708,"statfs":{"total":95995035648,"available":95965908992,"internally_reserved":0,"allocated":1454080,"data_stored":932812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234052,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":535398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430281,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1650688,"data_stored":1132313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593808,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":530676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790038,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37408,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707744,"statfs":{"total":95995035648,"available":95956729856,"internally_reserved":0,"allocated":2052096,"data_stored":1516738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953564,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":536169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33040,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712112,"statfs":{"total":95995035648,"available":95961202688,"internally_reserved":0,"allocated":1839104,"data_stored":1317862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378726,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1134592,"data_stored":601475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542253,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":599904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":532064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":806912,"data_stored":787276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":395784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":76088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:03:49.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:03:50.269 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:50.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.924+0000 7fe04b200640 1 -- 172.21.15.39:0/51057723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044103b60 msgr2=0x7fe044103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.924+0000 7fe04b200640 1 --2- 172.21.15.39:0/51057723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044103b60 0x7fe044103f70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe034009900 tx=0x7fe03402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 -- 172.21.15.39:0/51057723 shutdown_connections 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 --2- 172.21.15.39:0/51057723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044109df0 0x7fe044110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 --2- 172.21.15.39:0/51057723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 0x7fe0441098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 --2- 172.21.15.39:0/51057723 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044103b60 0x7fe044103f70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 -- 172.21.15.39:0/51057723 >> 172.21.15.39:0/51057723 conn(0x7fe04407b850 msgr2=0x7fe04407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 -- 172.21.15.39:0/51057723 shutdown_connections 2024-09-21T01:03:50.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.925+0000 7fe04b200640 1 -- 172.21.15.39:0/51057723 wait complete. 2024-09-21T01:03:50.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.926+0000 7fe04b200640 1 Processor -- start 2024-09-21T01:03:50.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.926+0000 7fe04b200640 1 -- start start 2024-09-21T01:03:50.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe04b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044103b60 0x7fe04419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe04b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 0x7fe04419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe04b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044109df0 0x7fe0441a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe04b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe044113ab0 con 0x7fe0441094a0 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe04b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe044113930 con 0x7fe044103b60 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe04b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe044113c30 con 0x7fe044109df0 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe043400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 0x7fe04419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe043e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044103b60 0x7fe04419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe043400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 0x7fe04419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57568/0 (socket says 172.21.15.39:57568) 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe043400640 1 -- 172.21.15.39:0/1464383139 learned_addr learned my addr 172.21.15.39:0/1464383139 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.927+0000 7fe049200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044109df0 0x7fe0441a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:50.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.928+0000 7fe043e00640 1 -- 172.21.15.39:0/1464383139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044109df0 msgr2=0x7fe0441a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:50.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.928+0000 7fe043e00640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044109df0 0x7fe0441a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:50.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.928+0000 7fe043e00640 1 -- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 msgr2=0x7fe04419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:50.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.928+0000 7fe043e00640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 0x7fe04419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:50.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.928+0000 7fe043e00640 1 -- 172.21.15.39:0/1464383139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0441a26b0 con 0x7fe044103b60 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.928+0000 7fe043e00640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044103b60 0x7fe04419c740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe03402a910 tx=0x7fe034031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.929+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03403f030 con 0x7fe044103b60 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.929+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0441a2940 con 0x7fe044103b60 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.929+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0340043d0 con 0x7fe044103b60 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.929+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe034035070 con 0x7fe044103b60 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.929+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0441a9100 con 0x7fe044103b60 2024-09-21T01:03:50.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.930+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe044103f70 con 0x7fe044103b60 2024-09-21T01:03:50.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.933+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fe034042030 con 0x7fe044103b60 2024-09-21T01:03:50.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.934+0000 7fe040c00640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe0280b65f0 0x7fe0280b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:50.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.934+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe034107e50 con 0x7fe044103b60 2024-09-21T01:03:50.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.934+0000 7fe043400640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe0280b65f0 0x7fe0280b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:50.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.935+0000 7fe043400640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe0280b65f0 0x7fe0280b8b00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe044108fe0 tx=0x7fe030005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:50.936+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe0340ce080 con 0x7fe044103b60 2024-09-21T01:03:51.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.135+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe044107a80 con 0x7fe0280b65f0 2024-09-21T01:03:51.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.141+0000 7fe040c00640 1 -- 172.21.15.39:0/1464383139 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340533 (secure 0 0 0) 0x7fe044107a80 con 0x7fe0280b65f0 2024-09-21T01:03:51.147 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe0280b65f0 msgr2=0x7fe0280b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe0280b65f0 0x7fe0280b8b00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe044108fe0 tx=0x7fe030005f90 comp rx=0 tx=0).stop 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044103b60 msgr2=0x7fe04419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044103b60 0x7fe04419c740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe03402a910 tx=0x7fe034031a20 comp rx=0 tx=0).stop 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 shutdown_connections 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe044109df0 0x7fe0441a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0441094a0 0x7fe04419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe0280b65f0 0x7fe0280b8b00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:51.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 --2- 172.21.15.39:0/1464383139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe044103b60 0x7fe04419c740 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:51.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.145+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 >> 172.21.15.39:0/1464383139 conn(0x7fe04407b850 msgr2=0x7fe044107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:51.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.146+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 shutdown_connections 2024-09-21T01:03:51.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:51.146+0000 7fe04b200640 1 -- 172.21.15.39:0/1464383139 wait complete. 2024-09-21T01:03:51.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:51 smithi160 ceph-mon[34696]: pgmap v399: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:51.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:51 smithi039 ceph-mon[30415]: pgmap v399: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:51.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:51 smithi150 ceph-mon[34945]: pgmap v399: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:52.138 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-21T01:03:49.940946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1456246,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1213,"num_read_kb":852,"num_write":537,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":898,"ondisk_log_size":898,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373992,"kb_used_data":16244,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567832,"statfs":{"total":1151940427776,"available":1151557459968,"internally_reserved":0,"allocated":16633856,"data_stored":10374620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":791963,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":410,"num_read_kb":344,"num_write":233,"num_write_kb":2099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003869"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489181+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:02:10.489181+0000","last_peered":"2024-09-21T01:02:10.489181+0000","last_clean":"2024-09-21T01:02:10.489181+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:02:10.489181+0000","last_undegraded":"2024-09-21T01:02:10.489181+0000","last_fullsized":"2024-09-21T01:02:10.489181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296157+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:03:08.296157+0000","last_peered":"2024-09-21T01:03:08.296157+0000","last_clean":"2024-09-21T01:03:08.296157+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:03:08.296157+0000","last_undegraded":"2024-09-21T01:03:08.296157+0000","last_fullsized":"2024-09-21T01:03:08.296157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485309+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:02:10.485309+0000","last_peered":"2024-09-21T01:02:10.485309+0000","last_clean":"2024-09-21T01:02:10.485309+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:02:10.485309+0000","last_undegraded":"2024-09-21T01:02:10.485309+0000","last_fullsized":"2024-09-21T01:02:10.485309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.144022+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:02:11.144022+0000","last_peered":"2024-09-21T01:02:11.144022+0000","last_clean":"2024-09-21T01:02:11.144022+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:02:11.144022+0000","last_undegraded":"2024-09-21T01:02:11.144022+0000","last_fullsized":"2024-09-21T01:02:11.144022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.308007+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:03:08.308007+0000","last_peered":"2024-09-21T01:03:08.308007+0000","last_clean":"2024-09-21T01:03:08.308007+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:03:08.308007+0000","last_undegraded":"2024-09-21T01:03:08.308007+0000","last_fullsized":"2024-09-21T01:03:08.308007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487364+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:02:10.487364+0000","last_peered":"2024-09-21T01:02:10.487364+0000","last_clean":"2024-09-21T01:02:10.487364+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:02:10.487364+0000","last_undegraded":"2024-09-21T01:02:10.487364+0000","last_fullsized":"2024-09-21T01:02:10.487364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137270+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:02:11.137270+0000","last_peered":"2024-09-21T01:02:11.137270+0000","last_clean":"2024-09-21T01:02:11.137270+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:02:11.137270+0000","last_undegraded":"2024-09-21T01:02:11.137270+0000","last_fullsized":"2024-09-21T01:02:11.137270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307119+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:03:08.307119+0000","last_peered":"2024-09-21T01:03:08.307119+0000","last_clean":"2024-09-21T01:03:08.307119+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:03:08.307119+0000","last_undegraded":"2024-09-21T01:03:08.307119+0000","last_fullsized":"2024-09-21T01:03:08.307119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487404+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:02:10.487404+0000","last_peered":"2024-09-21T01:02:10.487404+0000","last_clean":"2024-09-21T01:02:10.487404+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:02:10.487404+0000","last_undegraded":"2024-09-21T01:02:10.487404+0000","last_fullsized":"2024-09-21T01:02:10.487404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137027+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:02:11.137027+0000","last_peered":"2024-09-21T01:02:11.137027+0000","last_clean":"2024-09-21T01:02:11.137027+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:02:11.137027+0000","last_undegraded":"2024-09-21T01:02:11.137027+0000","last_fullsized":"2024-09-21T01:02:11.137027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301310+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:03:08.301310+0000","last_peered":"2024-09-21T01:03:08.301310+0000","last_clean":"2024-09-21T01:03:08.301310+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:03:08.301310+0000","last_undegraded":"2024-09-21T01:03:08.301310+0000","last_fullsized":"2024-09-21T01:03:08.301310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487356+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:02:10.487356+0000","last_peered":"2024-09-21T01:02:10.487356+0000","last_clean":"2024-09-21T01:02:10.487356+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:02:10.487356+0000","last_undegraded":"2024-09-21T01:02:10.487356+0000","last_fullsized":"2024-09-21T01:02:10.487356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.306792+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:03:08.306792+0000","last_peered":"2024-09-21T01:03:08.306792+0000","last_clean":"2024-09-21T01:03:08.306792+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:03:08.306792+0000","last_undegraded":"2024-09-21T01:03:08.306792+0000","last_fullsized":"2024-09-21T01:03:08.306792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136891+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:02:11.136891+0000","last_peered":"2024-09-21T01:02:11.136891+0000","last_clean":"2024-09-21T01:02:11.136891+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:02:11.136891+0000","last_undegraded":"2024-09-21T01:02:11.136891+0000","last_fullsized":"2024-09-21T01:02:11.136891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488041+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:02:10.488041+0000","last_peered":"2024-09-21T01:02:10.488041+0000","last_clean":"2024-09-21T01:02:10.488041+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:02:10.488041+0000","last_undegraded":"2024-09-21T01:02:10.488041+0000","last_fullsized":"2024-09-21T01:02:10.488041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304480+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:03:08.304480+0000","last_peered":"2024-09-21T01:03:08.304480+0000","last_clean":"2024-09-21T01:03:08.304480+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:03:08.304480+0000","last_undegraded":"2024-09-21T01:03:08.304480+0000","last_fullsized":"2024-09-21T01:03:08.304480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485418+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:02:10.485418+0000","last_peered":"2024-09-21T01:02:10.485418+0000","last_clean":"2024-09-21T01:02:10.485418+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:02:10.485418+0000","last_undegraded":"2024-09-21T01:02:10.485418+0000","last_fullsized":"2024-09-21T01:02:10.485418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487939+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:02:10.487939+0000","last_peered":"2024-09-21T01:02:10.487939+0000","last_clean":"2024-09-21T01:02:10.487939+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:02:10.487939+0000","last_undegraded":"2024-09-21T01:02:10.487939+0000","last_fullsized":"2024-09-21T01:02:10.487939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304538+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:03:08.304538+0000","last_peered":"2024-09-21T01:03:08.304538+0000","last_clean":"2024-09-21T01:03:08.304538+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:03:08.304538+0000","last_undegraded":"2024-09-21T01:03:08.304538+0000","last_fullsized":"2024-09-21T01:03:08.304538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143393+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:02:11.143393+0000","last_peered":"2024-09-21T01:02:11.143393+0000","last_clean":"2024-09-21T01:02:11.143393+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:02:11.143393+0000","last_undegraded":"2024-09-21T01:02:11.143393+0000","last_fullsized":"2024-09-21T01:02:11.143393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488402+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:02:10.488402+0000","last_peered":"2024-09-21T01:02:10.488402+0000","last_clean":"2024-09-21T01:02:10.488402+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:02:10.488402+0000","last_undegraded":"2024-09-21T01:02:10.488402+0000","last_fullsized":"2024-09-21T01:02:10.488402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.254592+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:03:08.254592+0000","last_peered":"2024-09-21T01:03:08.254592+0000","last_clean":"2024-09-21T01:03:08.254592+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:03:08.254592+0000","last_undegraded":"2024-09-21T01:03:08.254592+0000","last_fullsized":"2024-09-21T01:03:08.254592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964583+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:02:10.964583+0000","last_peered":"2024-09-21T01:02:10.964583+0000","last_clean":"2024-09-21T01:02:10.964583+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:02:10.964583+0000","last_undegraded":"2024-09-21T01:02:10.964583+0000","last_fullsized":"2024-09-21T01:02:10.964583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488121+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:02:10.488121+0000","last_peered":"2024-09-21T01:02:10.488121+0000","last_clean":"2024-09-21T01:02:10.488121+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:02:10.488121+0000","last_undegraded":"2024-09-21T01:02:10.488121+0000","last_fullsized":"2024-09-21T01:02:10.488121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293749+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:03:08.293749+0000","last_peered":"2024-09-21T01:03:08.293749+0000","last_clean":"2024-09-21T01:03:08.293749+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:03:08.293749+0000","last_undegraded":"2024-09-21T01:03:08.293749+0000","last_fullsized":"2024-09-21T01:03:08.293749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488427+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:02:10.488427+0000","last_peered":"2024-09-21T01:02:10.488427+0000","last_clean":"2024-09-21T01:02:10.488427+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:02:10.488427+0000","last_undegraded":"2024-09-21T01:02:10.488427+0000","last_fullsized":"2024-09-21T01:02:10.488427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485629+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:02:10.485629+0000","last_peered":"2024-09-21T01:02:10.485629+0000","last_clean":"2024-09-21T01:02:10.485629+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:02:10.485629+0000","last_undegraded":"2024-09-21T01:02:10.485629+0000","last_fullsized":"2024-09-21T01:02:10.485629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301452+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:03:08.301452+0000","last_peered":"2024-09-21T01:03:08.301452+0000","last_clean":"2024-09-21T01:03:08.301452+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:03:08.301452+0000","last_undegraded":"2024-09-21T01:03:08.301452+0000","last_fullsized":"2024-09-21T01:03:08.301452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'97","reported_seq":231,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:47.291053+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:03:47.291053+0000","last_peered":"2024-09-21T01:03:47.291053+0000","last_clean":"2024-09-21T01:03:47.291053+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:03:47.291053+0000","last_undegraded":"2024-09-21T01:03:47.291053+0000","last_fullsized":"2024-09-21T01:03:47.291053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":178,"num_write_kb":1221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142973+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:02:11.142973+0000","last_peered":"2024-09-21T01:02:11.142973+0000","last_clean":"2024-09-21T01:02:11.142973+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:02:11.142973+0000","last_undegraded":"2024-09-21T01:02:11.142973+0000","last_fullsized":"2024-09-21T01:02:11.142973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296507+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:03:08.296507+0000","last_peered":"2024-09-21T01:03:08.296507+0000","last_clean":"2024-09-21T01:03:08.296507+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:03:08.296507+0000","last_undegraded":"2024-09-21T01:03:08.296507+0000","last_fullsized":"2024-09-21T01:03:08.296507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137262+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:02:11.137262+0000","last_peered":"2024-09-21T01:02:11.137262+0000","last_clean":"2024-09-21T01:02:11.137262+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:02:11.137262+0000","last_undegraded":"2024-09-21T01:02:11.137262+0000","last_fullsized":"2024-09-21T01:02:11.137262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316968+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:02:11.316968+0000","last_peered":"2024-09-21T01:02:11.316968+0000","last_clean":"2024-09-21T01:02:11.316968+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:02:11.316968+0000","last_undegraded":"2024-09-21T01:02:11.316968+0000","last_fullsized":"2024-09-21T01:02:11.316968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.287755+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:03:08.287755+0000","last_peered":"2024-09-21T01:03:08.287755+0000","last_clean":"2024-09-21T01:03:08.287755+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:03:08.287755+0000","last_undegraded":"2024-09-21T01:03:08.287755+0000","last_fullsized":"2024-09-21T01:03:08.287755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488662+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:02:10.488662+0000","last_peered":"2024-09-21T01:02:10.488662+0000","last_clean":"2024-09-21T01:02:10.488662+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:02:10.488662+0000","last_undegraded":"2024-09-21T01:02:10.488662+0000","last_fullsized":"2024-09-21T01:02:10.488662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136558+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:02:11.136558+0000","last_peered":"2024-09-21T01:02:11.136558+0000","last_clean":"2024-09-21T01:02:11.136558+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:02:11.136558+0000","last_undegraded":"2024-09-21T01:02:11.136558+0000","last_fullsized":"2024-09-21T01:02:11.136558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293464+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:03:08.293464+0000","last_peered":"2024-09-21T01:03:08.293464+0000","last_clean":"2024-09-21T01:03:08.293464+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:03:08.293464+0000","last_undegraded":"2024-09-21T01:03:08.293464+0000","last_fullsized":"2024-09-21T01:03:08.293464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490887+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:02:10.490887+0000","last_peered":"2024-09-21T01:02:10.490887+0000","last_clean":"2024-09-21T01:02:10.490887+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:02:10.490887+0000","last_undegraded":"2024-09-21T01:02:10.490887+0000","last_fullsized":"2024-09-21T01:02:10.490887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484822+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:02:10.484822+0000","last_peered":"2024-09-21T01:02:10.484822+0000","last_clean":"2024-09-21T01:02:10.484822+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:02:10.484822+0000","last_undegraded":"2024-09-21T01:02:10.484822+0000","last_fullsized":"2024-09-21T01:02:10.484822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.292886+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:03:08.292886+0000","last_peered":"2024-09-21T01:03:08.292886+0000","last_clean":"2024-09-21T01:03:08.292886+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:03:08.292886+0000","last_undegraded":"2024-09-21T01:03:08.292886+0000","last_fullsized":"2024-09-21T01:03:08.292886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:45.603143+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:03:45.603143+0000","last_peered":"2024-09-21T01:03:45.603143+0000","last_clean":"2024-09-21T01:03:45.603143+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:03:45.603143+0000","last_undegraded":"2024-09-21T01:03:45.603143+0000","last_fullsized":"2024-09-21T01:03:45.603143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485099+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:02:10.485099+0000","last_peered":"2024-09-21T01:02:10.485099+0000","last_clean":"2024-09-21T01:02:10.485099+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:02:10.485099+0000","last_undegraded":"2024-09-21T01:02:10.485099+0000","last_fullsized":"2024-09-21T01:02:10.485099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296516+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:03:08.296516+0000","last_peered":"2024-09-21T01:03:08.296516+0000","last_clean":"2024-09-21T01:03:08.296516+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:03:08.296516+0000","last_undegraded":"2024-09-21T01:03:08.296516+0000","last_fullsized":"2024-09-21T01:03:08.296516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490838+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:02:10.490838+0000","last_peered":"2024-09-21T01:02:10.490838+0000","last_clean":"2024-09-21T01:02:10.490838+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:02:10.490838+0000","last_undegraded":"2024-09-21T01:02:10.490838+0000","last_fullsized":"2024-09-21T01:02:10.490838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489999+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:02:10.489999+0000","last_peered":"2024-09-21T01:02:10.489999+0000","last_clean":"2024-09-21T01:02:10.489999+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:02:10.489999+0000","last_undegraded":"2024-09-21T01:02:10.489999+0000","last_fullsized":"2024-09-21T01:02:10.489999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301050+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:03:08.301050+0000","last_peered":"2024-09-21T01:03:08.301050+0000","last_clean":"2024-09-21T01:03:08.301050+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:03:08.301050+0000","last_undegraded":"2024-09-21T01:03:08.301050+0000","last_fullsized":"2024-09-21T01:03:08.301050+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487931+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:02:10.487931+0000","last_peered":"2024-09-21T01:02:10.487931+0000","last_clean":"2024-09-21T01:02:10.487931+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:02:10.487931+0000","last_undegraded":"2024-09-21T01:02:10.487931+0000","last_fullsized":"2024-09-21T01:02:10.487931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490885+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:02:10.490885+0000","last_peered":"2024-09-21T01:02:10.490885+0000","last_clean":"2024-09-21T01:02:10.490885+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:02:10.490885+0000","last_undegraded":"2024-09-21T01:02:10.490885+0000","last_fullsized":"2024-09-21T01:02:10.490885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490252+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:02:10.490252+0000","last_peered":"2024-09-21T01:02:10.490252+0000","last_clean":"2024-09-21T01:02:10.490252+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:02:10.490252+0000","last_undegraded":"2024-09-21T01:02:10.490252+0000","last_fullsized":"2024-09-21T01:02:10.490252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301604+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:03:08.301604+0000","last_peered":"2024-09-21T01:03:08.301604+0000","last_clean":"2024-09-21T01:03:08.301604+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:03:08.301604+0000","last_undegraded":"2024-09-21T01:03:08.301604+0000","last_fullsized":"2024-09-21T01:03:08.301604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485952+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:02:10.485952+0000","last_peered":"2024-09-21T01:02:10.485952+0000","last_clean":"2024-09-21T01:02:10.485952+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:02:10.485952+0000","last_undegraded":"2024-09-21T01:02:10.485952+0000","last_fullsized":"2024-09-21T01:02:10.485952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.965031+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:02:10.965031+0000","last_peered":"2024-09-21T01:02:10.965031+0000","last_clean":"2024-09-21T01:02:10.965031+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:02:10.965031+0000","last_undegraded":"2024-09-21T01:02:10.965031+0000","last_fullsized":"2024-09-21T01:02:10.965031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300921+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:03:08.300921+0000","last_peered":"2024-09-21T01:03:08.300921+0000","last_clean":"2024-09-21T01:03:08.300921+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:03:08.300921+0000","last_undegraded":"2024-09-21T01:03:08.300921+0000","last_fullsized":"2024-09-21T01:03:08.300921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'60","reported_seq":133,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:46.298866+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:03:46.298866+0000","last_peered":"2024-09-21T01:03:46.298866+0000","last_clean":"2024-09-21T01:03:46.298866+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:03:46.298866+0000","last_undegraded":"2024-09-21T01:03:46.298866+0000","last_fullsized":"2024-09-21T01:03:46.298866+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645580+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:02:10.645580+0000","last_peered":"2024-09-21T01:02:10.645580+0000","last_clean":"2024-09-21T01:02:10.645580+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:02:10.645580+0000","last_undegraded":"2024-09-21T01:02:10.645580+0000","last_fullsized":"2024-09-21T01:02:10.645580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302551+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:03:08.302551+0000","last_peered":"2024-09-21T01:03:08.302551+0000","last_clean":"2024-09-21T01:03:08.302551+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:03:08.302551+0000","last_undegraded":"2024-09-21T01:03:08.302551+0000","last_fullsized":"2024-09-21T01:03:08.302551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485106+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:02:10.485106+0000","last_peered":"2024-09-21T01:02:10.485106+0000","last_clean":"2024-09-21T01:02:10.485106+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:02:10.485106+0000","last_undegraded":"2024-09-21T01:02:10.485106+0000","last_fullsized":"2024-09-21T01:02:10.485106+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489060+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:02:10.489060+0000","last_peered":"2024-09-21T01:02:10.489060+0000","last_clean":"2024-09-21T01:02:10.489060+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:02:10.489060+0000","last_undegraded":"2024-09-21T01:02:10.489060+0000","last_fullsized":"2024-09-21T01:02:10.489060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290221+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:03:08.290221+0000","last_peered":"2024-09-21T01:03:08.290221+0000","last_clean":"2024-09-21T01:03:08.290221+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:03:08.290221+0000","last_undegraded":"2024-09-21T01:03:08.290221+0000","last_fullsized":"2024-09-21T01:03:08.290221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:25.005750+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:03:25.005750+0000","last_peered":"2024-09-21T01:03:25.005750+0000","last_clean":"2024-09-21T01:03:25.005750+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:03:25.005750+0000","last_undegraded":"2024-09-21T01:03:25.005750+0000","last_fullsized":"2024-09-21T01:03:25.005750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489786+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:02:10.489786+0000","last_peered":"2024-09-21T01:02:10.489786+0000","last_clean":"2024-09-21T01:02:10.489786+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:02:10.489786+0000","last_undegraded":"2024-09-21T01:02:10.489786+0000","last_fullsized":"2024-09-21T01:02:10.489786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295651+0000","last_change":"2024-09-21T01:02:10.966004+0000","last_active":"2024-09-21T01:03:08.295651+0000","last_peered":"2024-09-21T01:03:08.295651+0000","last_clean":"2024-09-21T01:03:08.295651+0000","last_became_active":"2024-09-21T01:02:10.965645+0000","last_became_peered":"2024-09-21T01:02:10.965645+0000","last_unstale":"2024-09-21T01:03:08.295651+0000","last_undegraded":"2024-09-21T01:03:08.295651+0000","last_fullsized":"2024-09-21T01:03:08.295651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:12.070017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136620+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:02:11.136620+0000","last_peered":"2024-09-21T01:02:11.136620+0000","last_clean":"2024-09-21T01:02:11.136620+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:02:11.136620+0000","last_undegraded":"2024-09-21T01:02:11.136620+0000","last_fullsized":"2024-09-21T01:02:11.136620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485028+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:02:10.485028+0000","last_peered":"2024-09-21T01:02:10.485028+0000","last_clean":"2024-09-21T01:02:10.485028+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:02:10.485028+0000","last_undegraded":"2024-09-21T01:02:10.485028+0000","last_fullsized":"2024-09-21T01:02:10.485028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300660+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:03:08.300660+0000","last_peered":"2024-09-21T01:03:08.300660+0000","last_clean":"2024-09-21T01:03:08.300660+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:03:08.300660+0000","last_undegraded":"2024-09-21T01:03:08.300660+0000","last_fullsized":"2024-09-21T01:03:08.300660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:44.602219+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:03:44.602219+0000","last_peered":"2024-09-21T01:03:44.602219+0000","last_clean":"2024-09-21T01:03:44.602219+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:03:44.602219+0000","last_undegraded":"2024-09-21T01:03:44.602219+0000","last_fullsized":"2024-09-21T01:03:44.602219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1431,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489132+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:02:10.489132+0000","last_peered":"2024-09-21T01:02:10.489132+0000","last_clean":"2024-09-21T01:02:10.489132+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:02:10.489132+0000","last_undegraded":"2024-09-21T01:02:10.489132+0000","last_fullsized":"2024-09-21T01:02:10.489132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304638+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:03:08.304638+0000","last_peered":"2024-09-21T01:03:08.304638+0000","last_clean":"2024-09-21T01:03:08.304638+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:03:08.304638+0000","last_undegraded":"2024-09-21T01:03:08.304638+0000","last_fullsized":"2024-09-21T01:03:08.304638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489455+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:02:10.489455+0000","last_peered":"2024-09-21T01:02:10.489455+0000","last_clean":"2024-09-21T01:02:10.489455+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:02:10.489455+0000","last_undegraded":"2024-09-21T01:02:10.489455+0000","last_fullsized":"2024-09-21T01:02:10.489455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316922+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:02:11.316922+0000","last_peered":"2024-09-21T01:02:11.316922+0000","last_clean":"2024-09-21T01:02:11.316922+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:02:11.316922+0000","last_undegraded":"2024-09-21T01:02:11.316922+0000","last_fullsized":"2024-09-21T01:02:11.316922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298020+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:03:08.298020+0000","last_peered":"2024-09-21T01:03:08.298020+0000","last_clean":"2024-09-21T01:03:08.298020+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:03:08.298020+0000","last_undegraded":"2024-09-21T01:03:08.298020+0000","last_fullsized":"2024-09-21T01:03:08.298020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485567+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:02:10.485567+0000","last_peered":"2024-09-21T01:02:10.485567+0000","last_clean":"2024-09-21T01:02:10.485567+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:02:10.485567+0000","last_undegraded":"2024-09-21T01:02:10.485567+0000","last_fullsized":"2024-09-21T01:02:10.485567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489683+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:02:10.489683+0000","last_peered":"2024-09-21T01:02:10.489683+0000","last_clean":"2024-09-21T01:02:10.489683+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:02:10.489683+0000","last_undegraded":"2024-09-21T01:02:10.489683+0000","last_fullsized":"2024-09-21T01:02:10.489683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297147+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:03:08.297147+0000","last_peered":"2024-09-21T01:03:08.297147+0000","last_clean":"2024-09-21T01:03:08.297147+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:03:08.297147+0000","last_undegraded":"2024-09-21T01:03:08.297147+0000","last_fullsized":"2024-09-21T01:03:08.297147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:49.582968+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:03:49.582968+0000","last_peered":"2024-09-21T01:03:49.582968+0000","last_clean":"2024-09-21T01:03:49.582968+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:03:49.582968+0000","last_undegraded":"2024-09-21T01:03:49.582968+0000","last_fullsized":"2024-09-21T01:03:49.582968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73667,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":271,"num_read_kb":208,"num_write":42,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488195+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:02:10.488195+0000","last_peered":"2024-09-21T01:02:10.488195+0000","last_clean":"2024-09-21T01:02:10.488195+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:02:10.488195+0000","last_undegraded":"2024-09-21T01:02:10.488195+0000","last_fullsized":"2024-09-21T01:02:10.488195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.305890+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:03:08.305890+0000","last_peered":"2024-09-21T01:03:08.305890+0000","last_clean":"2024-09-21T01:03:08.305890+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:03:08.305890+0000","last_undegraded":"2024-09-21T01:03:08.305890+0000","last_fullsized":"2024-09-21T01:03:08.305890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489820+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:02:10.489820+0000","last_peered":"2024-09-21T01:02:10.489820+0000","last_clean":"2024-09-21T01:02:10.489820+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:02:10.489820+0000","last_undegraded":"2024-09-21T01:02:10.489820+0000","last_fullsized":"2024-09-21T01:02:10.489820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487710+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:02:10.487710+0000","last_peered":"2024-09-21T01:02:10.487710+0000","last_clean":"2024-09-21T01:02:10.487710+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:02:10.487710+0000","last_undegraded":"2024-09-21T01:02:10.487710+0000","last_fullsized":"2024-09-21T01:02:10.487710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297611+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:03:08.297611+0000","last_peered":"2024-09-21T01:03:08.297611+0000","last_clean":"2024-09-21T01:03:08.297611+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:03:08.297611+0000","last_undegraded":"2024-09-21T01:03:08.297611+0000","last_fullsized":"2024-09-21T01:03:08.297611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490763+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:02:10.490763+0000","last_peered":"2024-09-21T01:02:10.490763+0000","last_clean":"2024-09-21T01:02:10.490763+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:02:10.490763+0000","last_undegraded":"2024-09-21T01:02:10.490763+0000","last_fullsized":"2024-09-21T01:02:10.490763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487966+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:02:10.487966+0000","last_peered":"2024-09-21T01:02:10.487966+0000","last_clean":"2024-09-21T01:02:10.487966+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:02:10.487966+0000","last_undegraded":"2024-09-21T01:02:10.487966+0000","last_fullsized":"2024-09-21T01:02:10.487966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.305939+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:03:08.305939+0000","last_peered":"2024-09-21T01:03:08.305939+0000","last_clean":"2024-09-21T01:03:08.305939+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:03:08.305939+0000","last_undegraded":"2024-09-21T01:03:08.305939+0000","last_fullsized":"2024-09-21T01:03:08.305939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:48.581311+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:03:48.581311+0000","last_peered":"2024-09-21T01:03:48.581311+0000","last_clean":"2024-09-21T01:03:48.581311+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:03:48.581311+0000","last_undegraded":"2024-09-21T01:03:48.581311+0000","last_fullsized":"2024-09-21T01:03:48.581311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143140+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:02:11.143140+0000","last_peered":"2024-09-21T01:02:11.143140+0000","last_clean":"2024-09-21T01:02:11.143140+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:02:11.143140+0000","last_undegraded":"2024-09-21T01:02:11.143140+0000","last_fullsized":"2024-09-21T01:02:11.143140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645756+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:02:10.645756+0000","last_peered":"2024-09-21T01:02:10.645756+0000","last_clean":"2024-09-21T01:02:10.645756+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:02:10.645756+0000","last_undegraded":"2024-09-21T01:02:10.645756+0000","last_fullsized":"2024-09-21T01:02:10.645756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301916+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:03:08.301916+0000","last_peered":"2024-09-21T01:03:08.301916+0000","last_clean":"2024-09-21T01:03:08.301916+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:03:08.301916+0000","last_undegraded":"2024-09-21T01:03:08.301916+0000","last_fullsized":"2024-09-21T01:03:08.301916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489429+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:02:10.489429+0000","last_peered":"2024-09-21T01:02:10.489429+0000","last_clean":"2024-09-21T01:02:10.489429+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:02:10.489429+0000","last_undegraded":"2024-09-21T01:02:10.489429+0000","last_fullsized":"2024-09-21T01:02:10.489429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317166+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:02:11.317166+0000","last_peered":"2024-09-21T01:02:11.317166+0000","last_clean":"2024-09-21T01:02:11.317166+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:02:11.317166+0000","last_undegraded":"2024-09-21T01:02:11.317166+0000","last_fullsized":"2024-09-21T01:02:11.317166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297901+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:03:08.297901+0000","last_peered":"2024-09-21T01:03:08.297901+0000","last_clean":"2024-09-21T01:03:08.297901+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:03:08.297901+0000","last_undegraded":"2024-09-21T01:03:08.297901+0000","last_fullsized":"2024-09-21T01:03:08.297901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484277+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:02:10.484277+0000","last_peered":"2024-09-21T01:02:10.484277+0000","last_clean":"2024-09-21T01:02:10.484277+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:02:10.484277+0000","last_undegraded":"2024-09-21T01:02:10.484277+0000","last_fullsized":"2024-09-21T01:02:10.484277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517818+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:02:10.517818+0000","last_peered":"2024-09-21T01:02:10.517818+0000","last_clean":"2024-09-21T01:02:10.517818+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:02:10.517818+0000","last_undegraded":"2024-09-21T01:02:10.517818+0000","last_fullsized":"2024-09-21T01:02:10.517818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302502+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:03:08.302502+0000","last_peered":"2024-09-21T01:03:08.302502+0000","last_clean":"2024-09-21T01:03:08.302502+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:03:08.302502+0000","last_undegraded":"2024-09-21T01:03:08.302502+0000","last_fullsized":"2024-09-21T01:03:08.302502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485639+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:02:10.485639+0000","last_peered":"2024-09-21T01:02:10.485639+0000","last_clean":"2024-09-21T01:02:10.485639+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:02:10.485639+0000","last_undegraded":"2024-09-21T01:02:10.485639+0000","last_fullsized":"2024-09-21T01:02:10.485639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:45.577011+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:03:45.577011+0000","last_peered":"2024-09-21T01:03:45.577011+0000","last_clean":"2024-09-21T01:03:45.577011+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:03:45.577011+0000","last_undegraded":"2024-09-21T01:03:45.577011+0000","last_fullsized":"2024-09-21T01:03:45.577011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295018+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:03:08.295018+0000","last_peered":"2024-09-21T01:03:08.295018+0000","last_clean":"2024-09-21T01:03:08.295018+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:03:08.295018+0000","last_undegraded":"2024-09-21T01:03:08.295018+0000","last_fullsized":"2024-09-21T01:03:08.295018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485589+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:02:10.485589+0000","last_peered":"2024-09-21T01:02:10.485589+0000","last_clean":"2024-09-21T01:02:10.485589+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:02:10.485589+0000","last_undegraded":"2024-09-21T01:02:10.485589+0000","last_fullsized":"2024-09-21T01:02:10.485589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.294307+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:03:08.294307+0000","last_peered":"2024-09-21T01:03:08.294307+0000","last_clean":"2024-09-21T01:03:08.294307+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:03:08.294307+0000","last_undegraded":"2024-09-21T01:03:08.294307+0000","last_fullsized":"2024-09-21T01:03:08.294307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297962+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:03:08.297962+0000","last_peered":"2024-09-21T01:03:08.297962+0000","last_clean":"2024-09-21T01:03:08.297962+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:03:08.297962+0000","last_undegraded":"2024-09-21T01:03:08.297962+0000","last_fullsized":"2024-09-21T01:03:08.297962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.276740+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:03:08.276740+0000","last_peered":"2024-09-21T01:03:08.276740+0000","last_clean":"2024-09-21T01:03:08.276740+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:03:08.276740+0000","last_undegraded":"2024-09-21T01:03:08.276740+0000","last_fullsized":"2024-09-21T01:03:08.276740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.299954+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:03:08.299954+0000","last_peered":"2024-09-21T01:03:08.299954+0000","last_clean":"2024-09-21T01:03:08.299954+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:03:08.299954+0000","last_undegraded":"2024-09-21T01:03:08.299954+0000","last_fullsized":"2024-09-21T01:03:08.299954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.303193+0000","last_change":"2024-09-21T01:02:10.647066+0000","last_active":"2024-09-21T01:03:08.303193+0000","last_peered":"2024-09-21T01:03:08.303193+0000","last_clean":"2024-09-21T01:03:08.303193+0000","last_became_active":"2024-09-21T01:02:10.646487+0000","last_became_peered":"2024-09-21T01:02:10.646487+0000","last_unstale":"2024-09-21T01:03:08.303193+0000","last_undegraded":"2024-09-21T01:03:08.303193+0000","last_fullsized":"2024-09-21T01:03:08.303193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:19:28.445094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296459+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:03:08.296459+0000","last_peered":"2024-09-21T01:03:08.296459+0000","last_clean":"2024-09-21T01:03:08.296459+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:03:08.296459+0000","last_undegraded":"2024-09-21T01:03:08.296459+0000","last_fullsized":"2024-09-21T01:03:08.296459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295660+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:03:08.295660+0000","last_peered":"2024-09-21T01:03:08.295660+0000","last_clean":"2024-09-21T01:03:08.295660+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:03:08.295660+0000","last_undegraded":"2024-09-21T01:03:08.295660+0000","last_fullsized":"2024-09-21T01:03:08.295660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.291803+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:03:08.291803+0000","last_peered":"2024-09-21T01:03:08.291803+0000","last_clean":"2024-09-21T01:03:08.291803+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:03:08.291803+0000","last_undegraded":"2024-09-21T01:03:08.291803+0000","last_fullsized":"2024-09-21T01:03:08.291803+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295455+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:03:08.295455+0000","last_peered":"2024-09-21T01:03:08.295455+0000","last_clean":"2024-09-21T01:03:08.295455+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:03:08.295455+0000","last_undegraded":"2024-09-21T01:03:08.295455+0000","last_fullsized":"2024-09-21T01:03:08.295455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293894+0000","last_change":"2024-09-21T01:02:11.318295+0000","last_active":"2024-09-21T01:03:08.293894+0000","last_peered":"2024-09-21T01:03:08.293894+0000","last_clean":"2024-09-21T01:03:08.293894+0000","last_became_active":"2024-09-21T01:02:11.317912+0000","last_became_peered":"2024-09-21T01:02:11.317912+0000","last_unstale":"2024-09-21T01:03:08.293894+0000","last_undegraded":"2024-09-21T01:03:08.293894+0000","last_fullsized":"2024-09-21T01:03:08.293894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:53:47.063340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307810+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:03:08.307810+0000","last_peered":"2024-09-21T01:03:08.307810+0000","last_clean":"2024-09-21T01:03:08.307810+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:03:08.307810+0000","last_undegraded":"2024-09-21T01:03:08.307810+0000","last_fullsized":"2024-09-21T01:03:08.307810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298761+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:03:08.298761+0000","last_peered":"2024-09-21T01:03:08.298761+0000","last_clean":"2024-09-21T01:03:08.298761+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:03:08.298761+0000","last_undegraded":"2024-09-21T01:03:08.298761+0000","last_fullsized":"2024-09-21T01:03:08.298761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296634+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:03:08.296634+0000","last_peered":"2024-09-21T01:03:08.296634+0000","last_clean":"2024-09-21T01:03:08.296634+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:03:08.296634+0000","last_undegraded":"2024-09-21T01:03:08.296634+0000","last_fullsized":"2024-09-21T01:03:08.296634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290097+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:03:08.290097+0000","last_peered":"2024-09-21T01:03:08.290097+0000","last_clean":"2024-09-21T01:03:08.290097+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:03:08.290097+0000","last_undegraded":"2024-09-21T01:03:08.290097+0000","last_fullsized":"2024-09-21T01:03:08.290097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.299853+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:03:08.299853+0000","last_peered":"2024-09-21T01:03:08.299853+0000","last_clean":"2024-09-21T01:03:08.299853+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:03:08.299853+0000","last_undegraded":"2024-09-21T01:03:08.299853+0000","last_fullsized":"2024-09-21T01:03:08.299853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307223+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:03:08.307223+0000","last_peered":"2024-09-21T01:03:08.307223+0000","last_clean":"2024-09-21T01:03:08.307223+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:03:08.307223+0000","last_undegraded":"2024-09-21T01:03:08.307223+0000","last_fullsized":"2024-09-21T01:03:08.307223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307652+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:03:08.307652+0000","last_peered":"2024-09-21T01:03:08.307652+0000","last_clean":"2024-09-21T01:03:08.307652+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:03:08.307652+0000","last_undegraded":"2024-09-21T01:03:08.307652+0000","last_fullsized":"2024-09-21T01:03:08.307652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298544+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:03:08.298544+0000","last_peered":"2024-09-21T01:03:08.298544+0000","last_clean":"2024-09-21T01:03:08.298544+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:03:08.298544+0000","last_undegraded":"2024-09-21T01:03:08.298544+0000","last_fullsized":"2024-09-21T01:03:08.298544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304775+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:03:08.304775+0000","last_peered":"2024-09-21T01:03:08.304775+0000","last_clean":"2024-09-21T01:03:08.304775+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:03:08.304775+0000","last_undegraded":"2024-09-21T01:03:08.304775+0000","last_fullsized":"2024-09-21T01:03:08.304775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300375+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:03:08.300375+0000","last_peered":"2024-09-21T01:03:08.300375+0000","last_clean":"2024-09-21T01:03:08.300375+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:03:08.300375+0000","last_undegraded":"2024-09-21T01:03:08.300375+0000","last_fullsized":"2024-09-21T01:03:08.300375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297218+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:03:08.297218+0000","last_peered":"2024-09-21T01:03:08.297218+0000","last_clean":"2024-09-21T01:03:08.297218+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:03:08.297218+0000","last_undegraded":"2024-09-21T01:03:08.297218+0000","last_fullsized":"2024-09-21T01:03:08.297218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302497+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:03:08.302497+0000","last_peered":"2024-09-21T01:03:08.302497+0000","last_clean":"2024-09-21T01:03:08.302497+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:03:08.302497+0000","last_undegraded":"2024-09-21T01:03:08.302497+0000","last_fullsized":"2024-09-21T01:03:08.302497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290792+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:03:08.290792+0000","last_peered":"2024-09-21T01:03:08.290792+0000","last_clean":"2024-09-21T01:03:08.290792+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:03:08.290792+0000","last_undegraded":"2024-09-21T01:03:08.290792+0000","last_fullsized":"2024-09-21T01:03:08.290792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.306589+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:03:08.306589+0000","last_peered":"2024-09-21T01:03:08.306589+0000","last_clean":"2024-09-21T01:03:08.306589+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:03:08.306589+0000","last_undegraded":"2024-09-21T01:03:08.306589+0000","last_fullsized":"2024-09-21T01:03:08.306589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301598+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:03:08.301598+0000","last_peered":"2024-09-21T01:03:08.301598+0000","last_clean":"2024-09-21T01:03:08.301598+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:03:08.301598+0000","last_undegraded":"2024-09-21T01:03:08.301598+0000","last_fullsized":"2024-09-21T01:03:08.301598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.294226+0000","last_change":"2024-09-21T01:02:11.138284+0000","last_active":"2024-09-21T01:03:08.294226+0000","last_peered":"2024-09-21T01:03:08.294226+0000","last_clean":"2024-09-21T01:03:08.294226+0000","last_became_active":"2024-09-21T01:02:11.137782+0000","last_became_peered":"2024-09-21T01:02:11.137782+0000","last_unstale":"2024-09-21T01:03:08.294226+0000","last_undegraded":"2024-09-21T01:03:08.294226+0000","last_fullsized":"2024-09-21T01:03:08.294226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:57.753015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298298+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:03:08.298298+0000","last_peered":"2024-09-21T01:03:08.298298+0000","last_clean":"2024-09-21T01:03:08.298298+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:03:08.298298+0000","last_undegraded":"2024-09-21T01:03:08.298298+0000","last_fullsized":"2024-09-21T01:03:08.298298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.292151+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:03:08.292151+0000","last_peered":"2024-09-21T01:03:08.292151+0000","last_clean":"2024-09-21T01:03:08.292151+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:03:08.292151+0000","last_undegraded":"2024-09-21T01:03:08.292151+0000","last_fullsized":"2024-09-21T01:03:08.292151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297492+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:03:08.297492+0000","last_peered":"2024-09-21T01:03:08.297492+0000","last_clean":"2024-09-21T01:03:08.297492+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:03:08.297492+0000","last_undegraded":"2024-09-21T01:03:08.297492+0000","last_fullsized":"2024-09-21T01:03:08.297492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307974+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:03:08.307974+0000","last_peered":"2024-09-21T01:03:08.307974+0000","last_clean":"2024-09-21T01:03:08.307974+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:03:08.307974+0000","last_undegraded":"2024-09-21T01:03:08.307974+0000","last_fullsized":"2024-09-21T01:03:08.307974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.303566+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:03:08.303566+0000","last_peered":"2024-09-21T01:03:08.303566+0000","last_clean":"2024-09-21T01:03:08.303566+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:03:08.303566+0000","last_undegraded":"2024-09-21T01:03:08.303566+0000","last_fullsized":"2024-09-21T01:03:08.303566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307001+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:03:08.307001+0000","last_peered":"2024-09-21T01:03:08.307001+0000","last_clean":"2024-09-21T01:03:08.307001+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:03:08.307001+0000","last_undegraded":"2024-09-21T01:03:08.307001+0000","last_fullsized":"2024-09-21T01:03:08.307001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":865644,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1132,"num_read_kb":783,"num_write":412,"num_write_kb":2231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2433024,"data_stored":2203716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678067,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":535286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874296,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29176,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93715976,"statfs":{"total":95995035648,"available":95965159424,"internally_reserved":0,"allocated":2142208,"data_stored":1594914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037823,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28444,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716708,"statfs":{"total":95995035648,"available":95965908992,"internally_reserved":0,"allocated":1454080,"data_stored":932812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234052,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":535398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430282,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1650688,"data_stored":1132313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593808,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":530676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790038,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37408,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707744,"statfs":{"total":95995035648,"available":95956729856,"internally_reserved":0,"allocated":2052096,"data_stored":1516738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953564,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":536169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33040,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712112,"statfs":{"total":95995035648,"available":95961202688,"internally_reserved":0,"allocated":1839104,"data_stored":1317862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378727,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1138688,"data_stored":606135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542254,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1114112,"data_stored":604253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":532064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":73667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":806912,"data_stored":787276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":395784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":469864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:03:52.143 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T01:03:52.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:03:52.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:52 smithi160 ceph-mon[34696]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:52.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:52 smithi039 ceph-mon[30415]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:52.544 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:52.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:52 smithi150 ceph-mon[34945]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 -- 172.21.15.39:0/474059078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f981094b0 msgr2=0x7f3f981098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/474059078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f981094b0 0x7f3f981098c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009900 tx=0x7f3f8402a400 comp rx=0 tx=0).stop 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 -- 172.21.15.39:0/474059078 shutdown_connections 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/474059078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 0x7f3f98110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/474059078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f981094b0 0x7f3f981098c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/474059078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f98103b70 0x7f3f98103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.282+0000 7f3f9fc00640 1 -- 172.21.15.39:0/474059078 >> 172.21.15.39:0/474059078 conn(0x7f3f980ff5e0 msgr2=0x7f3f98101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:53.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.283+0000 7f3f9fc00640 1 -- 172.21.15.39:0/474059078 shutdown_connections 2024-09-21T01:03:53.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.283+0000 7f3f9fc00640 1 -- 172.21.15.39:0/474059078 wait complete. 2024-09-21T01:03:53.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 Processor -- start 2024-09-21T01:03:53.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 -- start start 2024-09-21T01:03:53.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 0x7f3f9819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:53.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f981094b0 0x7f3f9819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:53.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 0x7f3f981a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:53.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3f98113b10 con 0x7f3f98103b70 2024-09-21T01:03:53.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f98113990 con 0x7f3f98109e00 2024-09-21T01:03:53.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.284+0000 7f3f9fc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3f98113c90 con 0x7f3f981094b0 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 0x7f3f9819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f97e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f981094b0 0x7f3f9819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 0x7f3f9819c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58502/0 (socket says 172.21.15.39:58502) 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 -- 172.21.15.39:0/3947157536 learned_addr learned my addr 172.21.15.39:0/3947157536 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f97e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f981094b0 0x7f3f9819cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39318/0 (socket says 172.21.15.39:39318) 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9dc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 0x7f3f981a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 -- 172.21.15.39:0/3947157536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f981094b0 msgr2=0x7f3f9819cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f981094b0 0x7f3f9819cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 -- 172.21.15.39:0/3947157536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 msgr2=0x7f3f981a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 0x7f3f981a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9d200640 1 -- 172.21.15.39:0/3947157536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f981a2630 con 0x7f3f98103b70 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.285+0000 7f3f9dc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 0x7f3f981a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.286+0000 7f3f9d200640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 0x7f3f9819c6c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00ea60 tx=0x7f3f8c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.286+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c00cde0 con 0x7f3f98103b70 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.286+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8c0035f0 con 0x7f3f98103b70 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.286+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f981a2920 con 0x7f3f98103b70 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.286+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c010690 con 0x7f3f98103b70 2024-09-21T01:03:53.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.286+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f981a9080 con 0x7f3f98103b70 2024-09-21T01:03:53.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.288+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f98103f80 con 0x7f3f98103b70 2024-09-21T01:03:53.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.290+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f3f8c0037a0 con 0x7f3f98103b70 2024-09-21T01:03:53.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.292+0000 7f3f95600640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:53.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.292+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3f8c0e7cb0 con 0x7f3f98103b70 2024-09-21T01:03:53.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.292+0000 7f3f97e00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:53.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.293+0000 7f3f97e00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f8402a9f0 tx=0x7f3f84005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:53.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.296+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3f8c0adee0 con 0x7f3f98103b70 2024-09-21T01:03:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:53 smithi160 ceph-mon[34696]: pgmap v400: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:53.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:03:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:53 smithi039 ceph-mon[30415]: pgmap v400: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:53.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:03:53.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.493+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f9810f8e0 con 0x7f3f800b65f0 2024-09-21T01:03:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.500+0000 7f3f95600640 1 -- 172.21.15.39:0/3947157536 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340537 (secure 0 0 0) 0x7f3f9810f8e0 con 0x7f3f800b65f0 2024-09-21T01:03:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:53.506 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T01:03:53.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 msgr2=0x7f3f800b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:53.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f8402a9f0 tx=0x7f3f84005a20 comp rx=0 tx=0).stop 2024-09-21T01:03:53.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 msgr2=0x7f3f9819c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 0x7f3f9819c6c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00ea60 tx=0x7f3f8c00ef20 comp rx=0 tx=0).stop 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 shutdown_connections 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3f800b65f0 0x7f3f800b8b00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f98109e00 0x7f3f981a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3f981094b0 0x7f3f9819cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 --2- 172.21.15.39:0/3947157536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f98103b70 0x7f3f9819c6c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 >> 172.21.15.39:0/3947157536 conn(0x7f3f980ff5e0 msgr2=0x7f3f98106af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 shutdown_connections 2024-09-21T01:03:53.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:53.506+0000 7f3f9fc00640 1 -- 172.21.15.39:0/3947157536 wait complete. 2024-09-21T01:03:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:53 smithi150 ceph-mon[34945]: pgmap v400: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:03:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:03:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:03:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:03:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:54 smithi160 ceph-mon[34696]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:54.396 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:03:54.396 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:03:54.396 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:03:54.396 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:54 smithi039 ceph-mon[30415]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:54.433 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-21T01:03:51.941465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1456246,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1225,"num_read_kb":854,"num_write":537,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":898,"ondisk_log_size":898,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374376,"kb_used_data":16628,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567448,"statfs":{"total":1151940427776,"available":1151557066752,"internally_reserved":0,"allocated":17027072,"data_stored":10767836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":791963,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":407,"num_read_kb":336,"num_write":233,"num_write_kb":2099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003697"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489181+0000","last_change":"2024-09-21T01:00:39.430167+0000","last_active":"2024-09-21T01:02:10.489181+0000","last_peered":"2024-09-21T01:02:10.489181+0000","last_clean":"2024-09-21T01:02:10.489181+0000","last_became_active":"2024-09-21T01:00:39.429801+0000","last_became_peered":"2024-09-21T01:00:39.429801+0000","last_unstale":"2024-09-21T01:02:10.489181+0000","last_undegraded":"2024-09-21T01:02:10.489181+0000","last_fullsized":"2024-09-21T01:02:10.489181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:07:43.150895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296157+0000","last_change":"2024-09-21T01:00:42.498078+0000","last_active":"2024-09-21T01:03:08.296157+0000","last_peered":"2024-09-21T01:03:08.296157+0000","last_clean":"2024-09-21T01:03:08.296157+0000","last_became_active":"2024-09-21T01:00:42.497701+0000","last_became_peered":"2024-09-21T01:00:42.497701+0000","last_unstale":"2024-09-21T01:03:08.296157+0000","last_undegraded":"2024-09-21T01:03:08.296157+0000","last_fullsized":"2024-09-21T01:03:08.296157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.315892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485309+0000","last_change":"2024-09-21T01:01:10.124135+0000","last_active":"2024-09-21T01:02:10.485309+0000","last_peered":"2024-09-21T01:02:10.485309+0000","last_clean":"2024-09-21T01:02:10.485309+0000","last_became_active":"2024-09-21T01:01:10.123600+0000","last_became_peered":"2024-09-21T01:01:10.123600+0000","last_unstale":"2024-09-21T01:02:10.485309+0000","last_undegraded":"2024-09-21T01:02:10.485309+0000","last_fullsized":"2024-09-21T01:02:10.485309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:40:29.655509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.144022+0000","last_change":"2024-09-21T01:00:39.451482+0000","last_active":"2024-09-21T01:02:11.144022+0000","last_peered":"2024-09-21T01:02:11.144022+0000","last_clean":"2024-09-21T01:02:11.144022+0000","last_became_active":"2024-09-21T01:00:39.451069+0000","last_became_peered":"2024-09-21T01:00:39.451069+0000","last_unstale":"2024-09-21T01:02:11.144022+0000","last_undegraded":"2024-09-21T01:02:11.144022+0000","last_fullsized":"2024-09-21T01:02:11.144022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:46.756187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.308007+0000","last_change":"2024-09-21T01:00:42.507940+0000","last_active":"2024-09-21T01:03:08.308007+0000","last_peered":"2024-09-21T01:03:08.308007+0000","last_clean":"2024-09-21T01:03:08.308007+0000","last_became_active":"2024-09-21T01:00:42.507587+0000","last_became_peered":"2024-09-21T01:00:42.507587+0000","last_unstale":"2024-09-21T01:03:08.308007+0000","last_undegraded":"2024-09-21T01:03:08.308007+0000","last_fullsized":"2024-09-21T01:03:08.308007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:34:00.956534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487364+0000","last_change":"2024-09-21T01:00:52.600537+0000","last_active":"2024-09-21T01:02:10.487364+0000","last_peered":"2024-09-21T01:02:10.487364+0000","last_clean":"2024-09-21T01:02:10.487364+0000","last_became_active":"2024-09-21T01:00:52.600100+0000","last_became_peered":"2024-09-21T01:00:52.600100+0000","last_unstale":"2024-09-21T01:02:10.487364+0000","last_undegraded":"2024-09-21T01:02:10.487364+0000","last_fullsized":"2024-09-21T01:02:10.487364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:34.699886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137270+0000","last_change":"2024-09-21T01:00:39.436922+0000","last_active":"2024-09-21T01:02:11.137270+0000","last_peered":"2024-09-21T01:02:11.137270+0000","last_clean":"2024-09-21T01:02:11.137270+0000","last_became_active":"2024-09-21T01:00:39.435573+0000","last_became_peered":"2024-09-21T01:00:39.435573+0000","last_unstale":"2024-09-21T01:02:11.137270+0000","last_undegraded":"2024-09-21T01:02:11.137270+0000","last_fullsized":"2024-09-21T01:02:11.137270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:20.166399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307119+0000","last_change":"2024-09-21T01:00:42.491964+0000","last_active":"2024-09-21T01:03:08.307119+0000","last_peered":"2024-09-21T01:03:08.307119+0000","last_clean":"2024-09-21T01:03:08.307119+0000","last_became_active":"2024-09-21T01:00:42.489333+0000","last_became_peered":"2024-09-21T01:00:42.489333+0000","last_unstale":"2024-09-21T01:03:08.307119+0000","last_undegraded":"2024-09-21T01:03:08.307119+0000","last_fullsized":"2024-09-21T01:03:08.307119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:05:35.949611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487404+0000","last_change":"2024-09-21T01:00:52.591288+0000","last_active":"2024-09-21T01:02:10.487404+0000","last_peered":"2024-09-21T01:02:10.487404+0000","last_clean":"2024-09-21T01:02:10.487404+0000","last_became_active":"2024-09-21T01:00:52.590593+0000","last_became_peered":"2024-09-21T01:00:52.590593+0000","last_unstale":"2024-09-21T01:02:10.487404+0000","last_undegraded":"2024-09-21T01:02:10.487404+0000","last_fullsized":"2024-09-21T01:02:10.487404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:28:57.340396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137027+0000","last_change":"2024-09-21T01:00:39.432615+0000","last_active":"2024-09-21T01:02:11.137027+0000","last_peered":"2024-09-21T01:02:11.137027+0000","last_clean":"2024-09-21T01:02:11.137027+0000","last_became_active":"2024-09-21T01:00:39.432245+0000","last_became_peered":"2024-09-21T01:00:39.432245+0000","last_unstale":"2024-09-21T01:02:11.137027+0000","last_undegraded":"2024-09-21T01:02:11.137027+0000","last_fullsized":"2024-09-21T01:02:11.137027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:48.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301310+0000","last_change":"2024-09-21T01:00:42.500674+0000","last_active":"2024-09-21T01:03:08.301310+0000","last_peered":"2024-09-21T01:03:08.301310+0000","last_clean":"2024-09-21T01:03:08.301310+0000","last_became_active":"2024-09-21T01:00:42.500292+0000","last_became_peered":"2024-09-21T01:00:42.500292+0000","last_unstale":"2024-09-21T01:03:08.301310+0000","last_undegraded":"2024-09-21T01:03:08.301310+0000","last_fullsized":"2024-09-21T01:03:08.301310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:58.930265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487356+0000","last_change":"2024-09-21T01:00:52.592754+0000","last_active":"2024-09-21T01:02:10.487356+0000","last_peered":"2024-09-21T01:02:10.487356+0000","last_clean":"2024-09-21T01:02:10.487356+0000","last_became_active":"2024-09-21T01:00:52.591690+0000","last_became_peered":"2024-09-21T01:00:52.591690+0000","last_unstale":"2024-09-21T01:02:10.487356+0000","last_undegraded":"2024-09-21T01:02:10.487356+0000","last_fullsized":"2024-09-21T01:02:10.487356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:47.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.306792+0000","last_change":"2024-09-21T01:00:42.503924+0000","last_active":"2024-09-21T01:03:08.306792+0000","last_peered":"2024-09-21T01:03:08.306792+0000","last_clean":"2024-09-21T01:03:08.306792+0000","last_became_active":"2024-09-21T01:00:42.503490+0000","last_became_peered":"2024-09-21T01:00:42.503490+0000","last_unstale":"2024-09-21T01:03:08.306792+0000","last_undegraded":"2024-09-21T01:03:08.306792+0000","last_fullsized":"2024-09-21T01:03:08.306792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:16.446856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136891+0000","last_change":"2024-09-21T01:00:39.436074+0000","last_active":"2024-09-21T01:02:11.136891+0000","last_peered":"2024-09-21T01:02:11.136891+0000","last_clean":"2024-09-21T01:02:11.136891+0000","last_became_active":"2024-09-21T01:00:39.435453+0000","last_became_peered":"2024-09-21T01:00:39.435453+0000","last_unstale":"2024-09-21T01:02:11.136891+0000","last_undegraded":"2024-09-21T01:02:11.136891+0000","last_fullsized":"2024-09-21T01:02:11.136891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:30.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488041+0000","last_change":"2024-09-21T01:00:52.588993+0000","last_active":"2024-09-21T01:02:10.488041+0000","last_peered":"2024-09-21T01:02:10.488041+0000","last_clean":"2024-09-21T01:02:10.488041+0000","last_became_active":"2024-09-21T01:00:52.587324+0000","last_became_peered":"2024-09-21T01:00:52.587324+0000","last_unstale":"2024-09-21T01:02:10.488041+0000","last_undegraded":"2024-09-21T01:02:10.488041+0000","last_fullsized":"2024-09-21T01:02:10.488041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:15:37.695553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304480+0000","last_change":"2024-09-21T01:00:42.496924+0000","last_active":"2024-09-21T01:03:08.304480+0000","last_peered":"2024-09-21T01:03:08.304480+0000","last_clean":"2024-09-21T01:03:08.304480+0000","last_became_active":"2024-09-21T01:00:42.495879+0000","last_became_peered":"2024-09-21T01:00:42.495879+0000","last_unstale":"2024-09-21T01:03:08.304480+0000","last_undegraded":"2024-09-21T01:03:08.304480+0000","last_fullsized":"2024-09-21T01:03:08.304480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:08:54.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485418+0000","last_change":"2024-09-21T01:00:39.433489+0000","last_active":"2024-09-21T01:02:10.485418+0000","last_peered":"2024-09-21T01:02:10.485418+0000","last_clean":"2024-09-21T01:02:10.485418+0000","last_became_active":"2024-09-21T01:00:39.433104+0000","last_became_peered":"2024-09-21T01:00:39.433104+0000","last_unstale":"2024-09-21T01:02:10.485418+0000","last_undegraded":"2024-09-21T01:02:10.485418+0000","last_fullsized":"2024-09-21T01:02:10.485418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:39.234528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487939+0000","last_change":"2024-09-21T01:00:52.587760+0000","last_active":"2024-09-21T01:02:10.487939+0000","last_peered":"2024-09-21T01:02:10.487939+0000","last_clean":"2024-09-21T01:02:10.487939+0000","last_became_active":"2024-09-21T01:00:52.586931+0000","last_became_peered":"2024-09-21T01:00:52.586931+0000","last_unstale":"2024-09-21T01:02:10.487939+0000","last_undegraded":"2024-09-21T01:02:10.487939+0000","last_fullsized":"2024-09-21T01:02:10.487939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:32.362568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304538+0000","last_change":"2024-09-21T01:00:42.495067+0000","last_active":"2024-09-21T01:03:08.304538+0000","last_peered":"2024-09-21T01:03:08.304538+0000","last_clean":"2024-09-21T01:03:08.304538+0000","last_became_active":"2024-09-21T01:00:42.491140+0000","last_became_peered":"2024-09-21T01:00:42.491140+0000","last_unstale":"2024-09-21T01:03:08.304538+0000","last_undegraded":"2024-09-21T01:03:08.304538+0000","last_fullsized":"2024-09-21T01:03:08.304538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:58.639280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143393+0000","last_change":"2024-09-21T01:00:39.429762+0000","last_active":"2024-09-21T01:02:11.143393+0000","last_peered":"2024-09-21T01:02:11.143393+0000","last_clean":"2024-09-21T01:02:11.143393+0000","last_became_active":"2024-09-21T01:00:39.429305+0000","last_became_peered":"2024-09-21T01:00:39.429305+0000","last_unstale":"2024-09-21T01:02:11.143393+0000","last_undegraded":"2024-09-21T01:02:11.143393+0000","last_fullsized":"2024-09-21T01:02:11.143393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:21.654542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488402+0000","last_change":"2024-09-21T01:00:52.597251+0000","last_active":"2024-09-21T01:02:10.488402+0000","last_peered":"2024-09-21T01:02:10.488402+0000","last_clean":"2024-09-21T01:02:10.488402+0000","last_became_active":"2024-09-21T01:00:52.596778+0000","last_became_peered":"2024-09-21T01:00:52.596778+0000","last_unstale":"2024-09-21T01:02:10.488402+0000","last_undegraded":"2024-09-21T01:02:10.488402+0000","last_fullsized":"2024-09-21T01:02:10.488402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:35:08.064948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.254592+0000","last_change":"2024-09-21T01:00:42.488358+0000","last_active":"2024-09-21T01:03:08.254592+0000","last_peered":"2024-09-21T01:03:08.254592+0000","last_clean":"2024-09-21T01:03:08.254592+0000","last_became_active":"2024-09-21T01:00:42.487887+0000","last_became_peered":"2024-09-21T01:00:42.487887+0000","last_unstale":"2024-09-21T01:03:08.254592+0000","last_undegraded":"2024-09-21T01:03:08.254592+0000","last_fullsized":"2024-09-21T01:03:08.254592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:41.223748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.964583+0000","last_change":"2024-09-21T01:01:09.690508+0000","last_active":"2024-09-21T01:02:10.964583+0000","last_peered":"2024-09-21T01:02:10.964583+0000","last_clean":"2024-09-21T01:02:10.964583+0000","last_became_active":"2024-09-21T01:01:09.690175+0000","last_became_peered":"2024-09-21T01:01:09.690175+0000","last_unstale":"2024-09-21T01:02:10.964583+0000","last_undegraded":"2024-09-21T01:02:10.964583+0000","last_fullsized":"2024-09-21T01:02:10.964583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:41:23.533800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488121+0000","last_change":"2024-09-21T01:00:52.591895+0000","last_active":"2024-09-21T01:02:10.488121+0000","last_peered":"2024-09-21T01:02:10.488121+0000","last_clean":"2024-09-21T01:02:10.488121+0000","last_became_active":"2024-09-21T01:00:52.591432+0000","last_became_peered":"2024-09-21T01:00:52.591432+0000","last_unstale":"2024-09-21T01:02:10.488121+0000","last_undegraded":"2024-09-21T01:02:10.488121+0000","last_fullsized":"2024-09-21T01:02:10.488121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:11:56.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293749+0000","last_change":"2024-09-21T01:01:10.124125+0000","last_active":"2024-09-21T01:03:08.293749+0000","last_peered":"2024-09-21T01:03:08.293749+0000","last_clean":"2024-09-21T01:03:08.293749+0000","last_became_active":"2024-09-21T01:01:10.123627+0000","last_became_peered":"2024-09-21T01:01:10.123627+0000","last_unstale":"2024-09-21T01:03:08.293749+0000","last_undegraded":"2024-09-21T01:03:08.293749+0000","last_fullsized":"2024-09-21T01:03:08.293749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:03:37.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488427+0000","last_change":"2024-09-21T01:00:39.429922+0000","last_active":"2024-09-21T01:02:10.488427+0000","last_peered":"2024-09-21T01:02:10.488427+0000","last_clean":"2024-09-21T01:02:10.488427+0000","last_became_active":"2024-09-21T01:00:39.429533+0000","last_became_peered":"2024-09-21T01:00:39.429533+0000","last_unstale":"2024-09-21T01:02:10.488427+0000","last_undegraded":"2024-09-21T01:02:10.488427+0000","last_fullsized":"2024-09-21T01:02:10.488427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:28:59.321291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485629+0000","last_change":"2024-09-21T01:00:52.587688+0000","last_active":"2024-09-21T01:02:10.485629+0000","last_peered":"2024-09-21T01:02:10.485629+0000","last_clean":"2024-09-21T01:02:10.485629+0000","last_became_active":"2024-09-21T01:00:52.587181+0000","last_became_peered":"2024-09-21T01:00:52.587181+0000","last_unstale":"2024-09-21T01:02:10.485629+0000","last_undegraded":"2024-09-21T01:02:10.485629+0000","last_fullsized":"2024-09-21T01:02:10.485629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:59.690121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301452+0000","last_change":"2024-09-21T01:00:42.498760+0000","last_active":"2024-09-21T01:03:08.301452+0000","last_peered":"2024-09-21T01:03:08.301452+0000","last_clean":"2024-09-21T01:03:08.301452+0000","last_became_active":"2024-09-21T01:00:42.497928+0000","last_became_peered":"2024-09-21T01:00:42.497928+0000","last_unstale":"2024-09-21T01:03:08.301452+0000","last_undegraded":"2024-09-21T01:03:08.301452+0000","last_fullsized":"2024-09-21T01:03:08.301452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:29:17.311129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'97","reported_seq":231,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:47.291053+0000","last_change":"2024-09-21T01:00:39.430285+0000","last_active":"2024-09-21T01:03:47.291053+0000","last_peered":"2024-09-21T01:03:47.291053+0000","last_clean":"2024-09-21T01:03:47.291053+0000","last_became_active":"2024-09-21T01:00:39.429854+0000","last_became_peered":"2024-09-21T01:00:39.429854+0000","last_unstale":"2024-09-21T01:03:47.291053+0000","last_undegraded":"2024-09-21T01:03:47.291053+0000","last_fullsized":"2024-09-21T01:03:47.291053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:59:06.036187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":178,"num_write_kb":1221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.142973+0000","last_change":"2024-09-21T01:00:52.587706+0000","last_active":"2024-09-21T01:02:11.142973+0000","last_peered":"2024-09-21T01:02:11.142973+0000","last_clean":"2024-09-21T01:02:11.142973+0000","last_became_active":"2024-09-21T01:00:52.587186+0000","last_became_peered":"2024-09-21T01:00:52.587186+0000","last_unstale":"2024-09-21T01:02:11.142973+0000","last_undegraded":"2024-09-21T01:02:11.142973+0000","last_fullsized":"2024-09-21T01:02:11.142973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:48:14.960249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296507+0000","last_change":"2024-09-21T01:00:42.498836+0000","last_active":"2024-09-21T01:03:08.296507+0000","last_peered":"2024-09-21T01:03:08.296507+0000","last_clean":"2024-09-21T01:03:08.296507+0000","last_became_active":"2024-09-21T01:00:42.496210+0000","last_became_peered":"2024-09-21T01:00:42.496210+0000","last_unstale":"2024-09-21T01:03:08.296507+0000","last_undegraded":"2024-09-21T01:03:08.296507+0000","last_fullsized":"2024-09-21T01:03:08.296507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:20:49.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.137262+0000","last_change":"2024-09-21T01:00:39.451567+0000","last_active":"2024-09-21T01:02:11.137262+0000","last_peered":"2024-09-21T01:02:11.137262+0000","last_clean":"2024-09-21T01:02:11.137262+0000","last_became_active":"2024-09-21T01:00:39.451161+0000","last_became_peered":"2024-09-21T01:00:39.451161+0000","last_unstale":"2024-09-21T01:02:11.137262+0000","last_undegraded":"2024-09-21T01:02:11.137262+0000","last_fullsized":"2024-09-21T01:02:11.137262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:41.465794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316968+0000","last_change":"2024-09-21T01:00:52.600680+0000","last_active":"2024-09-21T01:02:11.316968+0000","last_peered":"2024-09-21T01:02:11.316968+0000","last_clean":"2024-09-21T01:02:11.316968+0000","last_became_active":"2024-09-21T01:00:52.600139+0000","last_became_peered":"2024-09-21T01:00:52.600139+0000","last_unstale":"2024-09-21T01:02:11.316968+0000","last_undegraded":"2024-09-21T01:02:11.316968+0000","last_fullsized":"2024-09-21T01:02:11.316968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:35:55.693887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.287755+0000","last_change":"2024-09-21T01:00:42.495435+0000","last_active":"2024-09-21T01:03:08.287755+0000","last_peered":"2024-09-21T01:03:08.287755+0000","last_clean":"2024-09-21T01:03:08.287755+0000","last_became_active":"2024-09-21T01:00:42.495097+0000","last_became_peered":"2024-09-21T01:00:42.495097+0000","last_unstale":"2024-09-21T01:03:08.287755+0000","last_undegraded":"2024-09-21T01:03:08.287755+0000","last_fullsized":"2024-09-21T01:03:08.287755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:34:08.058697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488662+0000","last_change":"2024-09-21T01:00:39.452451+0000","last_active":"2024-09-21T01:02:10.488662+0000","last_peered":"2024-09-21T01:02:10.488662+0000","last_clean":"2024-09-21T01:02:10.488662+0000","last_became_active":"2024-09-21T01:00:39.451941+0000","last_became_peered":"2024-09-21T01:00:39.451941+0000","last_unstale":"2024-09-21T01:02:10.488662+0000","last_undegraded":"2024-09-21T01:02:10.488662+0000","last_fullsized":"2024-09-21T01:02:10.488662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:08.119204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136558+0000","last_change":"2024-09-21T01:00:52.587184+0000","last_active":"2024-09-21T01:02:11.136558+0000","last_peered":"2024-09-21T01:02:11.136558+0000","last_clean":"2024-09-21T01:02:11.136558+0000","last_became_active":"2024-09-21T01:00:52.586705+0000","last_became_peered":"2024-09-21T01:00:52.586705+0000","last_unstale":"2024-09-21T01:02:11.136558+0000","last_undegraded":"2024-09-21T01:02:11.136558+0000","last_fullsized":"2024-09-21T01:02:11.136558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:01.446737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293464+0000","last_change":"2024-09-21T01:00:42.490679+0000","last_active":"2024-09-21T01:03:08.293464+0000","last_peered":"2024-09-21T01:03:08.293464+0000","last_clean":"2024-09-21T01:03:08.293464+0000","last_became_active":"2024-09-21T01:00:42.490322+0000","last_became_peered":"2024-09-21T01:00:42.490322+0000","last_unstale":"2024-09-21T01:03:08.293464+0000","last_undegraded":"2024-09-21T01:03:08.293464+0000","last_fullsized":"2024-09-21T01:03:08.293464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:54:59.936558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490887+0000","last_change":"2024-09-21T01:00:39.433013+0000","last_active":"2024-09-21T01:02:10.490887+0000","last_peered":"2024-09-21T01:02:10.490887+0000","last_clean":"2024-09-21T01:02:10.490887+0000","last_became_active":"2024-09-21T01:00:39.432249+0000","last_became_peered":"2024-09-21T01:00:39.432249+0000","last_unstale":"2024-09-21T01:02:10.490887+0000","last_undegraded":"2024-09-21T01:02:10.490887+0000","last_fullsized":"2024-09-21T01:02:10.490887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:48.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484822+0000","last_change":"2024-09-21T01:00:52.603989+0000","last_active":"2024-09-21T01:02:10.484822+0000","last_peered":"2024-09-21T01:02:10.484822+0000","last_clean":"2024-09-21T01:02:10.484822+0000","last_became_active":"2024-09-21T01:00:52.601197+0000","last_became_peered":"2024-09-21T01:00:52.601197+0000","last_unstale":"2024-09-21T01:02:10.484822+0000","last_undegraded":"2024-09-21T01:02:10.484822+0000","last_fullsized":"2024-09-21T01:02:10.484822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:34.030099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.292886+0000","last_change":"2024-09-21T01:01:09.691316+0000","last_active":"2024-09-21T01:03:08.292886+0000","last_peered":"2024-09-21T01:03:08.292886+0000","last_clean":"2024-09-21T01:03:08.292886+0000","last_became_active":"2024-09-21T01:01:09.690835+0000","last_became_peered":"2024-09-21T01:01:09.690835+0000","last_unstale":"2024-09-21T01:03:08.292886+0000","last_undegraded":"2024-09-21T01:03:08.292886+0000","last_fullsized":"2024-09-21T01:03:08.292886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:59:24.123960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:45.603143+0000","last_change":"2024-09-21T01:00:39.452419+0000","last_active":"2024-09-21T01:03:45.603143+0000","last_peered":"2024-09-21T01:03:45.603143+0000","last_clean":"2024-09-21T01:03:45.603143+0000","last_became_active":"2024-09-21T01:00:39.451989+0000","last_became_peered":"2024-09-21T01:00:39.451989+0000","last_unstale":"2024-09-21T01:03:45.603143+0000","last_undegraded":"2024-09-21T01:03:45.603143+0000","last_fullsized":"2024-09-21T01:03:45.603143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:04:02.362139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485099+0000","last_change":"2024-09-21T01:00:52.595967+0000","last_active":"2024-09-21T01:02:10.485099+0000","last_peered":"2024-09-21T01:02:10.485099+0000","last_clean":"2024-09-21T01:02:10.485099+0000","last_became_active":"2024-09-21T01:00:52.595200+0000","last_became_peered":"2024-09-21T01:00:52.595200+0000","last_unstale":"2024-09-21T01:02:10.485099+0000","last_undegraded":"2024-09-21T01:02:10.485099+0000","last_fullsized":"2024-09-21T01:02:10.485099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:13.285311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296516+0000","last_change":"2024-09-21T01:00:42.492818+0000","last_active":"2024-09-21T01:03:08.296516+0000","last_peered":"2024-09-21T01:03:08.296516+0000","last_clean":"2024-09-21T01:03:08.296516+0000","last_became_active":"2024-09-21T01:00:42.492474+0000","last_became_peered":"2024-09-21T01:00:42.492474+0000","last_unstale":"2024-09-21T01:03:08.296516+0000","last_undegraded":"2024-09-21T01:03:08.296516+0000","last_fullsized":"2024-09-21T01:03:08.296516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:52:18.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490838+0000","last_change":"2024-09-21T01:00:39.430107+0000","last_active":"2024-09-21T01:02:10.490838+0000","last_peered":"2024-09-21T01:02:10.490838+0000","last_clean":"2024-09-21T01:02:10.490838+0000","last_became_active":"2024-09-21T01:00:39.429739+0000","last_became_peered":"2024-09-21T01:00:39.429739+0000","last_unstale":"2024-09-21T01:02:10.490838+0000","last_undegraded":"2024-09-21T01:02:10.490838+0000","last_fullsized":"2024-09-21T01:02:10.490838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:36:55.674494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489999+0000","last_change":"2024-09-21T01:00:52.601021+0000","last_active":"2024-09-21T01:02:10.489999+0000","last_peered":"2024-09-21T01:02:10.489999+0000","last_clean":"2024-09-21T01:02:10.489999+0000","last_became_active":"2024-09-21T01:00:52.600264+0000","last_became_peered":"2024-09-21T01:00:52.600264+0000","last_unstale":"2024-09-21T01:02:10.489999+0000","last_undegraded":"2024-09-21T01:02:10.489999+0000","last_fullsized":"2024-09-21T01:02:10.489999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:15.726075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301050+0000","last_change":"2024-09-21T01:00:42.488427+0000","last_active":"2024-09-21T01:03:08.301050+0000","last_peered":"2024-09-21T01:03:08.301050+0000","last_clean":"2024-09-21T01:03:08.301050+0000","last_became_active":"2024-09-21T01:00:42.483928+0000","last_became_peered":"2024-09-21T01:00:42.483928+0000","last_unstale":"2024-09-21T01:03:08.301050+0000","last_undegraded":"2024-09-21T01:03:08.301050+0000","last_fullsized":"2024-09-21T01:03:08.301050+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:51:02.050208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487931+0000","last_change":"2024-09-21T01:00:39.428742+0000","last_active":"2024-09-21T01:02:10.487931+0000","last_peered":"2024-09-21T01:02:10.487931+0000","last_clean":"2024-09-21T01:02:10.487931+0000","last_became_active":"2024-09-21T01:00:39.427873+0000","last_became_peered":"2024-09-21T01:00:39.427873+0000","last_unstale":"2024-09-21T01:02:10.487931+0000","last_undegraded":"2024-09-21T01:02:10.487931+0000","last_fullsized":"2024-09-21T01:02:10.487931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:17:38.366775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490885+0000","last_change":"2024-09-21T00:59:20.212472+0000","last_active":"2024-09-21T01:02:10.490885+0000","last_peered":"2024-09-21T01:02:10.490885+0000","last_clean":"2024-09-21T01:02:10.490885+0000","last_became_active":"2024-09-21T00:59:20.206752+0000","last_became_peered":"2024-09-21T00:59:20.206752+0000","last_unstale":"2024-09-21T01:02:10.490885+0000","last_undegraded":"2024-09-21T01:02:10.490885+0000","last_fullsized":"2024-09-21T01:02:10.490885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T00:55:23.188772+0000","last_clean_scrub_stamp":"2024-09-21T00:55:23.188772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:32:20.921067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490252+0000","last_change":"2024-09-21T01:01:10.124927+0000","last_active":"2024-09-21T01:02:10.490252+0000","last_peered":"2024-09-21T01:02:10.490252+0000","last_clean":"2024-09-21T01:02:10.490252+0000","last_became_active":"2024-09-21T01:01:10.124282+0000","last_became_peered":"2024-09-21T01:01:10.124282+0000","last_unstale":"2024-09-21T01:02:10.490252+0000","last_undegraded":"2024-09-21T01:02:10.490252+0000","last_fullsized":"2024-09-21T01:02:10.490252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:14.945661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301604+0000","last_change":"2024-09-21T01:00:42.499256+0000","last_active":"2024-09-21T01:03:08.301604+0000","last_peered":"2024-09-21T01:03:08.301604+0000","last_clean":"2024-09-21T01:03:08.301604+0000","last_became_active":"2024-09-21T01:00:42.498799+0000","last_became_peered":"2024-09-21T01:00:42.498799+0000","last_unstale":"2024-09-21T01:03:08.301604+0000","last_undegraded":"2024-09-21T01:03:08.301604+0000","last_fullsized":"2024-09-21T01:03:08.301604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:00:09.532787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485952+0000","last_change":"2024-09-21T01:00:39.453755+0000","last_active":"2024-09-21T01:02:10.485952+0000","last_peered":"2024-09-21T01:02:10.485952+0000","last_clean":"2024-09-21T01:02:10.485952+0000","last_became_active":"2024-09-21T01:00:39.453394+0000","last_became_peered":"2024-09-21T01:00:39.453394+0000","last_unstale":"2024-09-21T01:02:10.485952+0000","last_undegraded":"2024-09-21T01:02:10.485952+0000","last_fullsized":"2024-09-21T01:02:10.485952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:28.620246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.965031+0000","last_change":"2024-09-21T01:00:52.592628+0000","last_active":"2024-09-21T01:02:10.965031+0000","last_peered":"2024-09-21T01:02:10.965031+0000","last_clean":"2024-09-21T01:02:10.965031+0000","last_became_active":"2024-09-21T01:00:52.592049+0000","last_became_peered":"2024-09-21T01:00:52.592049+0000","last_unstale":"2024-09-21T01:02:10.965031+0000","last_undegraded":"2024-09-21T01:02:10.965031+0000","last_fullsized":"2024-09-21T01:02:10.965031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:32:40.349494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300921+0000","last_change":"2024-09-21T01:01:10.126851+0000","last_active":"2024-09-21T01:03:08.300921+0000","last_peered":"2024-09-21T01:03:08.300921+0000","last_clean":"2024-09-21T01:03:08.300921+0000","last_became_active":"2024-09-21T01:01:10.126426+0000","last_became_peered":"2024-09-21T01:01:10.126426+0000","last_unstale":"2024-09-21T01:03:08.300921+0000","last_undegraded":"2024-09-21T01:03:08.300921+0000","last_fullsized":"2024-09-21T01:03:08.300921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:49.443318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'60","reported_seq":135,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:46.804376+0000","last_change":"2024-09-21T01:00:39.427962+0000","last_active":"2024-09-21T01:03:46.804376+0000","last_peered":"2024-09-21T01:03:46.804376+0000","last_clean":"2024-09-21T01:03:46.804376+0000","last_became_active":"2024-09-21T01:00:39.427516+0000","last_became_peered":"2024-09-21T01:00:39.427516+0000","last_unstale":"2024-09-21T01:03:46.804376+0000","last_undegraded":"2024-09-21T01:03:46.804376+0000","last_fullsized":"2024-09-21T01:03:46.804376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:38:24.337646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645580+0000","last_change":"2024-09-21T01:00:52.587553+0000","last_active":"2024-09-21T01:02:10.645580+0000","last_peered":"2024-09-21T01:02:10.645580+0000","last_clean":"2024-09-21T01:02:10.645580+0000","last_became_active":"2024-09-21T01:00:52.587107+0000","last_became_peered":"2024-09-21T01:00:52.587107+0000","last_unstale":"2024-09-21T01:02:10.645580+0000","last_undegraded":"2024-09-21T01:02:10.645580+0000","last_fullsized":"2024-09-21T01:02:10.645580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:37.056396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302551+0000","last_change":"2024-09-21T01:00:42.497897+0000","last_active":"2024-09-21T01:03:08.302551+0000","last_peered":"2024-09-21T01:03:08.302551+0000","last_clean":"2024-09-21T01:03:08.302551+0000","last_became_active":"2024-09-21T01:00:42.497517+0000","last_became_peered":"2024-09-21T01:00:42.497517+0000","last_unstale":"2024-09-21T01:03:08.302551+0000","last_undegraded":"2024-09-21T01:03:08.302551+0000","last_fullsized":"2024-09-21T01:03:08.302551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:22.455404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485106+0000","last_change":"2024-09-21T01:00:39.435886+0000","last_active":"2024-09-21T01:02:10.485106+0000","last_peered":"2024-09-21T01:02:10.485106+0000","last_clean":"2024-09-21T01:02:10.485106+0000","last_became_active":"2024-09-21T01:00:39.435395+0000","last_became_peered":"2024-09-21T01:00:39.435395+0000","last_unstale":"2024-09-21T01:02:10.485106+0000","last_undegraded":"2024-09-21T01:02:10.485106+0000","last_fullsized":"2024-09-21T01:02:10.485106+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:37.112984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489060+0000","last_change":"2024-09-21T01:00:52.593147+0000","last_active":"2024-09-21T01:02:10.489060+0000","last_peered":"2024-09-21T01:02:10.489060+0000","last_clean":"2024-09-21T01:02:10.489060+0000","last_became_active":"2024-09-21T01:00:52.592306+0000","last_became_peered":"2024-09-21T01:00:52.592306+0000","last_unstale":"2024-09-21T01:02:10.489060+0000","last_undegraded":"2024-09-21T01:02:10.489060+0000","last_fullsized":"2024-09-21T01:02:10.489060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:01.833329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290221+0000","last_change":"2024-09-21T01:00:42.486987+0000","last_active":"2024-09-21T01:03:08.290221+0000","last_peered":"2024-09-21T01:03:08.290221+0000","last_clean":"2024-09-21T01:03:08.290221+0000","last_became_active":"2024-09-21T01:00:42.486045+0000","last_became_peered":"2024-09-21T01:00:42.486045+0000","last_unstale":"2024-09-21T01:03:08.290221+0000","last_undegraded":"2024-09-21T01:03:08.290221+0000","last_fullsized":"2024-09-21T01:03:08.290221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:06:01.734001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:25.005750+0000","last_change":"2024-09-21T01:01:09.694964+0000","last_active":"2024-09-21T01:03:25.005750+0000","last_peered":"2024-09-21T01:03:25.005750+0000","last_clean":"2024-09-21T01:03:25.005750+0000","last_became_active":"2024-09-21T01:01:09.690350+0000","last_became_peered":"2024-09-21T01:01:09.690350+0000","last_unstale":"2024-09-21T01:03:25.005750+0000","last_undegraded":"2024-09-21T01:03:25.005750+0000","last_fullsized":"2024-09-21T01:03:25.005750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:21:39.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489786+0000","last_change":"2024-09-21T01:00:52.599171+0000","last_active":"2024-09-21T01:02:10.489786+0000","last_peered":"2024-09-21T01:02:10.489786+0000","last_clean":"2024-09-21T01:02:10.489786+0000","last_became_active":"2024-09-21T01:00:52.598717+0000","last_became_peered":"2024-09-21T01:00:52.598717+0000","last_unstale":"2024-09-21T01:02:10.489786+0000","last_undegraded":"2024-09-21T01:02:10.489786+0000","last_fullsized":"2024-09-21T01:02:10.489786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:03.238245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295651+0000","last_change":"2024-09-21T01:02:10.966004+0000","last_active":"2024-09-21T01:03:08.295651+0000","last_peered":"2024-09-21T01:03:08.295651+0000","last_clean":"2024-09-21T01:03:08.295651+0000","last_became_active":"2024-09-21T01:02:10.965645+0000","last_became_peered":"2024-09-21T01:02:10.965645+0000","last_unstale":"2024-09-21T01:03:08.295651+0000","last_undegraded":"2024-09-21T01:03:08.295651+0000","last_fullsized":"2024-09-21T01:03:08.295651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:12.070017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.136620+0000","last_change":"2024-09-21T01:00:39.452733+0000","last_active":"2024-09-21T01:02:11.136620+0000","last_peered":"2024-09-21T01:02:11.136620+0000","last_clean":"2024-09-21T01:02:11.136620+0000","last_became_active":"2024-09-21T01:00:39.452312+0000","last_became_peered":"2024-09-21T01:00:39.452312+0000","last_unstale":"2024-09-21T01:02:11.136620+0000","last_undegraded":"2024-09-21T01:02:11.136620+0000","last_fullsized":"2024-09-21T01:02:11.136620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:13:20.887966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485028+0000","last_change":"2024-09-21T01:00:52.594508+0000","last_active":"2024-09-21T01:02:10.485028+0000","last_peered":"2024-09-21T01:02:10.485028+0000","last_clean":"2024-09-21T01:02:10.485028+0000","last_became_active":"2024-09-21T01:00:52.590033+0000","last_became_peered":"2024-09-21T01:00:52.590033+0000","last_unstale":"2024-09-21T01:02:10.485028+0000","last_undegraded":"2024-09-21T01:02:10.485028+0000","last_fullsized":"2024-09-21T01:02:10.485028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:13:16.180817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300660+0000","last_change":"2024-09-21T01:00:42.491279+0000","last_active":"2024-09-21T01:03:08.300660+0000","last_peered":"2024-09-21T01:03:08.300660+0000","last_clean":"2024-09-21T01:03:08.300660+0000","last_became_active":"2024-09-21T01:00:42.490936+0000","last_became_peered":"2024-09-21T01:00:42.490936+0000","last_unstale":"2024-09-21T01:03:08.300660+0000","last_undegraded":"2024-09-21T01:03:08.300660+0000","last_fullsized":"2024-09-21T01:03:08.300660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:47:57.467278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:49.610955+0000","last_change":"2024-09-21T01:00:39.428374+0000","last_active":"2024-09-21T01:03:49.610955+0000","last_peered":"2024-09-21T01:03:49.610955+0000","last_clean":"2024-09-21T01:03:49.610955+0000","last_became_active":"2024-09-21T01:00:39.427905+0000","last_became_peered":"2024-09-21T01:00:39.427905+0000","last_unstale":"2024-09-21T01:03:49.610955+0000","last_undegraded":"2024-09-21T01:03:49.610955+0000","last_fullsized":"2024-09-21T01:03:49.610955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:09.697221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1431,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489132+0000","last_change":"2024-09-21T01:00:52.592575+0000","last_active":"2024-09-21T01:02:10.489132+0000","last_peered":"2024-09-21T01:02:10.489132+0000","last_clean":"2024-09-21T01:02:10.489132+0000","last_became_active":"2024-09-21T01:00:52.592109+0000","last_became_peered":"2024-09-21T01:00:52.592109+0000","last_unstale":"2024-09-21T01:02:10.489132+0000","last_undegraded":"2024-09-21T01:02:10.489132+0000","last_fullsized":"2024-09-21T01:02:10.489132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:04.977919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304638+0000","last_change":"2024-09-21T01:00:42.507697+0000","last_active":"2024-09-21T01:03:08.304638+0000","last_peered":"2024-09-21T01:03:08.304638+0000","last_clean":"2024-09-21T01:03:08.304638+0000","last_became_active":"2024-09-21T01:00:42.507311+0000","last_became_peered":"2024-09-21T01:00:42.507311+0000","last_unstale":"2024-09-21T01:03:08.304638+0000","last_undegraded":"2024-09-21T01:03:08.304638+0000","last_fullsized":"2024-09-21T01:03:08.304638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:08.854249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489455+0000","last_change":"2024-09-21T01:00:39.431684+0000","last_active":"2024-09-21T01:02:10.489455+0000","last_peered":"2024-09-21T01:02:10.489455+0000","last_clean":"2024-09-21T01:02:10.489455+0000","last_became_active":"2024-09-21T01:00:39.431237+0000","last_became_peered":"2024-09-21T01:00:39.431237+0000","last_unstale":"2024-09-21T01:02:10.489455+0000","last_undegraded":"2024-09-21T01:02:10.489455+0000","last_fullsized":"2024-09-21T01:02:10.489455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:41:45.423441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.316922+0000","last_change":"2024-09-21T01:00:52.600676+0000","last_active":"2024-09-21T01:02:11.316922+0000","last_peered":"2024-09-21T01:02:11.316922+0000","last_clean":"2024-09-21T01:02:11.316922+0000","last_became_active":"2024-09-21T01:00:52.600182+0000","last_became_peered":"2024-09-21T01:00:52.600182+0000","last_unstale":"2024-09-21T01:02:11.316922+0000","last_undegraded":"2024-09-21T01:02:11.316922+0000","last_fullsized":"2024-09-21T01:02:11.316922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:31.995993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298020+0000","last_change":"2024-09-21T01:00:42.496234+0000","last_active":"2024-09-21T01:03:08.298020+0000","last_peered":"2024-09-21T01:03:08.298020+0000","last_clean":"2024-09-21T01:03:08.298020+0000","last_became_active":"2024-09-21T01:00:42.495912+0000","last_became_peered":"2024-09-21T01:00:42.495912+0000","last_unstale":"2024-09-21T01:03:08.298020+0000","last_undegraded":"2024-09-21T01:03:08.298020+0000","last_fullsized":"2024-09-21T01:03:08.298020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:05:16.391105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485567+0000","last_change":"2024-09-21T01:00:39.433607+0000","last_active":"2024-09-21T01:02:10.485567+0000","last_peered":"2024-09-21T01:02:10.485567+0000","last_clean":"2024-09-21T01:02:10.485567+0000","last_became_active":"2024-09-21T01:00:39.433266+0000","last_became_peered":"2024-09-21T01:00:39.433266+0000","last_unstale":"2024-09-21T01:02:10.485567+0000","last_undegraded":"2024-09-21T01:02:10.485567+0000","last_fullsized":"2024-09-21T01:02:10.485567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:49.436165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489683+0000","last_change":"2024-09-21T01:00:52.601630+0000","last_active":"2024-09-21T01:02:10.489683+0000","last_peered":"2024-09-21T01:02:10.489683+0000","last_clean":"2024-09-21T01:02:10.489683+0000","last_became_active":"2024-09-21T01:00:52.600206+0000","last_became_peered":"2024-09-21T01:00:52.600206+0000","last_unstale":"2024-09-21T01:02:10.489683+0000","last_undegraded":"2024-09-21T01:02:10.489683+0000","last_fullsized":"2024-09-21T01:02:10.489683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:57:22.578500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297147+0000","last_change":"2024-09-21T01:00:42.496186+0000","last_active":"2024-09-21T01:03:08.297147+0000","last_peered":"2024-09-21T01:03:08.297147+0000","last_clean":"2024-09-21T01:03:08.297147+0000","last_became_active":"2024-09-21T01:00:42.495256+0000","last_became_peered":"2024-09-21T01:00:42.495256+0000","last_unstale":"2024-09-21T01:03:08.297147+0000","last_undegraded":"2024-09-21T01:03:08.297147+0000","last_fullsized":"2024-09-21T01:03:08.297147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:18.871027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:49.582968+0000","last_change":"2024-09-21T01:00:39.432249+0000","last_active":"2024-09-21T01:03:49.582968+0000","last_peered":"2024-09-21T01:03:49.582968+0000","last_clean":"2024-09-21T01:03:49.582968+0000","last_became_active":"2024-09-21T01:00:39.431711+0000","last_became_peered":"2024-09-21T01:00:39.431711+0000","last_unstale":"2024-09-21T01:03:49.582968+0000","last_undegraded":"2024-09-21T01:03:49.582968+0000","last_fullsized":"2024-09-21T01:03:49.582968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:06:56.769068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73667,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":271,"num_read_kb":208,"num_write":42,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.488195+0000","last_change":"2024-09-21T01:00:52.590720+0000","last_active":"2024-09-21T01:02:10.488195+0000","last_peered":"2024-09-21T01:02:10.488195+0000","last_clean":"2024-09-21T01:02:10.488195+0000","last_became_active":"2024-09-21T01:00:52.589935+0000","last_became_peered":"2024-09-21T01:00:52.589935+0000","last_unstale":"2024-09-21T01:02:10.488195+0000","last_undegraded":"2024-09-21T01:02:10.488195+0000","last_fullsized":"2024-09-21T01:02:10.488195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:05.334407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.305890+0000","last_change":"2024-09-21T01:00:42.493380+0000","last_active":"2024-09-21T01:03:08.305890+0000","last_peered":"2024-09-21T01:03:08.305890+0000","last_clean":"2024-09-21T01:03:08.305890+0000","last_became_active":"2024-09-21T01:00:42.493032+0000","last_became_peered":"2024-09-21T01:00:42.493032+0000","last_unstale":"2024-09-21T01:03:08.305890+0000","last_undegraded":"2024-09-21T01:03:08.305890+0000","last_fullsized":"2024-09-21T01:03:08.305890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:15:45.873115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489820+0000","last_change":"2024-09-21T01:00:39.453806+0000","last_active":"2024-09-21T01:02:10.489820+0000","last_peered":"2024-09-21T01:02:10.489820+0000","last_clean":"2024-09-21T01:02:10.489820+0000","last_became_active":"2024-09-21T01:00:39.453050+0000","last_became_peered":"2024-09-21T01:00:39.453050+0000","last_unstale":"2024-09-21T01:02:10.489820+0000","last_undegraded":"2024-09-21T01:02:10.489820+0000","last_fullsized":"2024-09-21T01:02:10.489820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:03:45.825652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487710+0000","last_change":"2024-09-21T01:01:09.688957+0000","last_active":"2024-09-21T01:02:10.487710+0000","last_peered":"2024-09-21T01:02:10.487710+0000","last_clean":"2024-09-21T01:02:10.487710+0000","last_became_active":"2024-09-21T01:01:09.688507+0000","last_became_peered":"2024-09-21T01:01:09.688507+0000","last_unstale":"2024-09-21T01:02:10.487710+0000","last_undegraded":"2024-09-21T01:02:10.487710+0000","last_fullsized":"2024-09-21T01:02:10.487710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:21:50.301521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297611+0000","last_change":"2024-09-21T01:00:42.502674+0000","last_active":"2024-09-21T01:03:08.297611+0000","last_peered":"2024-09-21T01:03:08.297611+0000","last_clean":"2024-09-21T01:03:08.297611+0000","last_became_active":"2024-09-21T01:00:42.502343+0000","last_became_peered":"2024-09-21T01:00:42.502343+0000","last_unstale":"2024-09-21T01:03:08.297611+0000","last_undegraded":"2024-09-21T01:03:08.297611+0000","last_fullsized":"2024-09-21T01:03:08.297611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:09:33.174003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.490763+0000","last_change":"2024-09-21T01:00:39.435847+0000","last_active":"2024-09-21T01:02:10.490763+0000","last_peered":"2024-09-21T01:02:10.490763+0000","last_clean":"2024-09-21T01:02:10.490763+0000","last_became_active":"2024-09-21T01:00:39.435386+0000","last_became_peered":"2024-09-21T01:00:39.435386+0000","last_unstale":"2024-09-21T01:02:10.490763+0000","last_undegraded":"2024-09-21T01:02:10.490763+0000","last_fullsized":"2024-09-21T01:02:10.490763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:08:00.585925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.487966+0000","last_change":"2024-09-21T01:00:52.591099+0000","last_active":"2024-09-21T01:02:10.487966+0000","last_peered":"2024-09-21T01:02:10.487966+0000","last_clean":"2024-09-21T01:02:10.487966+0000","last_became_active":"2024-09-21T01:00:52.590555+0000","last_became_peered":"2024-09-21T01:00:52.590555+0000","last_unstale":"2024-09-21T01:02:10.487966+0000","last_undegraded":"2024-09-21T01:02:10.487966+0000","last_fullsized":"2024-09-21T01:02:10.487966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:14.969195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.305939+0000","last_change":"2024-09-21T01:00:42.498605+0000","last_active":"2024-09-21T01:03:08.305939+0000","last_peered":"2024-09-21T01:03:08.305939+0000","last_clean":"2024-09-21T01:03:08.305939+0000","last_became_active":"2024-09-21T01:00:42.498218+0000","last_became_peered":"2024-09-21T01:00:42.498218+0000","last_unstale":"2024-09-21T01:03:08.305939+0000","last_undegraded":"2024-09-21T01:03:08.305939+0000","last_fullsized":"2024-09-21T01:03:08.305939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:45:17.484046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:48.581311+0000","last_change":"2024-09-21T01:00:39.427772+0000","last_active":"2024-09-21T01:03:48.581311+0000","last_peered":"2024-09-21T01:03:48.581311+0000","last_clean":"2024-09-21T01:03:48.581311+0000","last_became_active":"2024-09-21T01:00:39.427183+0000","last_became_peered":"2024-09-21T01:00:39.427183+0000","last_unstale":"2024-09-21T01:03:48.581311+0000","last_undegraded":"2024-09-21T01:03:48.581311+0000","last_fullsized":"2024-09-21T01:03:48.581311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:21.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.143140+0000","last_change":"2024-09-21T01:00:52.593011+0000","last_active":"2024-09-21T01:02:11.143140+0000","last_peered":"2024-09-21T01:02:11.143140+0000","last_clean":"2024-09-21T01:02:11.143140+0000","last_became_active":"2024-09-21T01:00:52.592543+0000","last_became_peered":"2024-09-21T01:00:52.592543+0000","last_unstale":"2024-09-21T01:02:11.143140+0000","last_undegraded":"2024-09-21T01:02:11.143140+0000","last_fullsized":"2024-09-21T01:02:11.143140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:29:16.833883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.645756+0000","last_change":"2024-09-21T01:00:52.596115+0000","last_active":"2024-09-21T01:02:10.645756+0000","last_peered":"2024-09-21T01:02:10.645756+0000","last_clean":"2024-09-21T01:02:10.645756+0000","last_became_active":"2024-09-21T01:00:52.595648+0000","last_became_peered":"2024-09-21T01:00:52.595648+0000","last_unstale":"2024-09-21T01:02:10.645756+0000","last_undegraded":"2024-09-21T01:02:10.645756+0000","last_fullsized":"2024-09-21T01:02:10.645756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:17:41.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301916+0000","last_change":"2024-09-21T01:00:42.495225+0000","last_active":"2024-09-21T01:03:08.301916+0000","last_peered":"2024-09-21T01:03:08.301916+0000","last_clean":"2024-09-21T01:03:08.301916+0000","last_became_active":"2024-09-21T01:00:42.494858+0000","last_became_peered":"2024-09-21T01:00:42.494858+0000","last_unstale":"2024-09-21T01:03:08.301916+0000","last_undegraded":"2024-09-21T01:03:08.301916+0000","last_fullsized":"2024-09-21T01:03:08.301916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:19.215989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.489429+0000","last_change":"2024-09-21T01:00:39.429341+0000","last_active":"2024-09-21T01:02:10.489429+0000","last_peered":"2024-09-21T01:02:10.489429+0000","last_clean":"2024-09-21T01:02:10.489429+0000","last_became_active":"2024-09-21T01:00:39.428962+0000","last_became_peered":"2024-09-21T01:00:39.428962+0000","last_unstale":"2024-09-21T01:02:10.489429+0000","last_undegraded":"2024-09-21T01:02:10.489429+0000","last_fullsized":"2024-09-21T01:02:10.489429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:26.262642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:11.317166+0000","last_change":"2024-09-21T01:01:09.691179+0000","last_active":"2024-09-21T01:02:11.317166+0000","last_peered":"2024-09-21T01:02:11.317166+0000","last_clean":"2024-09-21T01:02:11.317166+0000","last_became_active":"2024-09-21T01:01:09.690602+0000","last_became_peered":"2024-09-21T01:01:09.690602+0000","last_unstale":"2024-09-21T01:02:11.317166+0000","last_undegraded":"2024-09-21T01:02:11.317166+0000","last_fullsized":"2024-09-21T01:02:11.317166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:43.442160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297901+0000","last_change":"2024-09-21T01:00:42.490290+0000","last_active":"2024-09-21T01:03:08.297901+0000","last_peered":"2024-09-21T01:03:08.297901+0000","last_clean":"2024-09-21T01:03:08.297901+0000","last_became_active":"2024-09-21T01:00:42.489955+0000","last_became_peered":"2024-09-21T01:00:42.489955+0000","last_unstale":"2024-09-21T01:03:08.297901+0000","last_undegraded":"2024-09-21T01:03:08.297901+0000","last_fullsized":"2024-09-21T01:03:08.297901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:15:59.868438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.484277+0000","last_change":"2024-09-21T01:00:39.430030+0000","last_active":"2024-09-21T01:02:10.484277+0000","last_peered":"2024-09-21T01:02:10.484277+0000","last_clean":"2024-09-21T01:02:10.484277+0000","last_became_active":"2024-09-21T01:00:39.429579+0000","last_became_peered":"2024-09-21T01:00:39.429579+0000","last_unstale":"2024-09-21T01:02:10.484277+0000","last_undegraded":"2024-09-21T01:02:10.484277+0000","last_fullsized":"2024-09-21T01:02:10.484277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:40:08.639583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.517818+0000","last_change":"2024-09-21T01:00:39.430810+0000","last_active":"2024-09-21T01:02:10.517818+0000","last_peered":"2024-09-21T01:02:10.517818+0000","last_clean":"2024-09-21T01:02:10.517818+0000","last_became_active":"2024-09-21T01:00:39.430319+0000","last_became_peered":"2024-09-21T01:00:39.430319+0000","last_unstale":"2024-09-21T01:02:10.517818+0000","last_undegraded":"2024-09-21T01:02:10.517818+0000","last_fullsized":"2024-09-21T01:02:10.517818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:47:43.592971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302502+0000","last_change":"2024-09-21T01:00:42.502103+0000","last_active":"2024-09-21T01:03:08.302502+0000","last_peered":"2024-09-21T01:03:08.302502+0000","last_clean":"2024-09-21T01:03:08.302502+0000","last_became_active":"2024-09-21T01:00:42.501726+0000","last_became_peered":"2024-09-21T01:00:42.501726+0000","last_unstale":"2024-09-21T01:03:08.302502+0000","last_undegraded":"2024-09-21T01:03:08.302502+0000","last_fullsized":"2024-09-21T01:03:08.302502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:38:34.981835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485639+0000","last_change":"2024-09-21T01:00:52.595861+0000","last_active":"2024-09-21T01:02:10.485639+0000","last_peered":"2024-09-21T01:02:10.485639+0000","last_clean":"2024-09-21T01:02:10.485639+0000","last_became_active":"2024-09-21T01:00:52.595311+0000","last_became_peered":"2024-09-21T01:00:52.595311+0000","last_unstale":"2024-09-21T01:02:10.485639+0000","last_undegraded":"2024-09-21T01:02:10.485639+0000","last_fullsized":"2024-09-21T01:02:10.485639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:46.465871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:50.585127+0000","last_change":"2024-09-21T01:00:39.435999+0000","last_active":"2024-09-21T01:03:50.585127+0000","last_peered":"2024-09-21T01:03:50.585127+0000","last_clean":"2024-09-21T01:03:50.585127+0000","last_became_active":"2024-09-21T01:00:39.435566+0000","last_became_peered":"2024-09-21T01:00:39.435566+0000","last_unstale":"2024-09-21T01:03:50.585127+0000","last_undegraded":"2024-09-21T01:03:50.585127+0000","last_fullsized":"2024-09-21T01:03:50.585127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:38.410260+0000","last_clean_scrub_stamp":"2024-09-21T01:00:38.410260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:04.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":135,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295018+0000","last_change":"2024-09-21T01:00:42.494576+0000","last_active":"2024-09-21T01:03:08.295018+0000","last_peered":"2024-09-21T01:03:08.295018+0000","last_clean":"2024-09-21T01:03:08.295018+0000","last_became_active":"2024-09-21T01:00:42.493714+0000","last_became_peered":"2024-09-21T01:00:42.493714+0000","last_unstale":"2024-09-21T01:03:08.295018+0000","last_undegraded":"2024-09-21T01:03:08.295018+0000","last_fullsized":"2024-09-21T01:03:08.295018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:37:13.458999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:02:10.485589+0000","last_change":"2024-09-21T01:00:52.593456+0000","last_active":"2024-09-21T01:02:10.485589+0000","last_peered":"2024-09-21T01:02:10.485589+0000","last_clean":"2024-09-21T01:02:10.485589+0000","last_became_active":"2024-09-21T01:00:52.592691+0000","last_became_peered":"2024-09-21T01:00:52.592691+0000","last_unstale":"2024-09-21T01:02:10.485589+0000","last_undegraded":"2024-09-21T01:02:10.485589+0000","last_fullsized":"2024-09-21T01:02:10.485589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:51.547720+0000","last_clean_scrub_stamp":"2024-09-21T01:00:51.547720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:41:44.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.294307+0000","last_change":"2024-09-21T01:00:42.496950+0000","last_active":"2024-09-21T01:03:08.294307+0000","last_peered":"2024-09-21T01:03:08.294307+0000","last_clean":"2024-09-21T01:03:08.294307+0000","last_became_active":"2024-09-21T01:00:42.491848+0000","last_became_peered":"2024-09-21T01:00:42.491848+0000","last_unstale":"2024-09-21T01:03:08.294307+0000","last_undegraded":"2024-09-21T01:03:08.294307+0000","last_fullsized":"2024-09-21T01:03:08.294307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:20:58.870695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297962+0000","last_change":"2024-09-21T01:00:42.498683+0000","last_active":"2024-09-21T01:03:08.297962+0000","last_peered":"2024-09-21T01:03:08.297962+0000","last_clean":"2024-09-21T01:03:08.297962+0000","last_became_active":"2024-09-21T01:00:42.497938+0000","last_became_peered":"2024-09-21T01:00:42.497938+0000","last_unstale":"2024-09-21T01:03:08.297962+0000","last_undegraded":"2024-09-21T01:03:08.297962+0000","last_fullsized":"2024-09-21T01:03:08.297962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:05:47.185063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.276740+0000","last_change":"2024-09-21T01:00:42.494286+0000","last_active":"2024-09-21T01:03:08.276740+0000","last_peered":"2024-09-21T01:03:08.276740+0000","last_clean":"2024-09-21T01:03:08.276740+0000","last_became_active":"2024-09-21T01:00:42.493897+0000","last_became_peered":"2024-09-21T01:00:42.493897+0000","last_unstale":"2024-09-21T01:03:08.276740+0000","last_undegraded":"2024-09-21T01:03:08.276740+0000","last_fullsized":"2024-09-21T01:03:08.276740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:16:22.367890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.299954+0000","last_change":"2024-09-21T01:00:42.504266+0000","last_active":"2024-09-21T01:03:08.299954+0000","last_peered":"2024-09-21T01:03:08.299954+0000","last_clean":"2024-09-21T01:03:08.299954+0000","last_became_active":"2024-09-21T01:00:42.503942+0000","last_became_peered":"2024-09-21T01:00:42.503942+0000","last_unstale":"2024-09-21T01:03:08.299954+0000","last_undegraded":"2024-09-21T01:03:08.299954+0000","last_fullsized":"2024-09-21T01:03:08.299954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:48:25.107519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.303193+0000","last_change":"2024-09-21T01:02:10.647066+0000","last_active":"2024-09-21T01:03:08.303193+0000","last_peered":"2024-09-21T01:03:08.303193+0000","last_clean":"2024-09-21T01:03:08.303193+0000","last_became_active":"2024-09-21T01:02:10.646487+0000","last_became_peered":"2024-09-21T01:02:10.646487+0000","last_unstale":"2024-09-21T01:03:08.303193+0000","last_undegraded":"2024-09-21T01:03:08.303193+0000","last_fullsized":"2024-09-21T01:03:08.303193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:19:28.445094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296459+0000","last_change":"2024-09-21T01:00:42.490102+0000","last_active":"2024-09-21T01:03:08.296459+0000","last_peered":"2024-09-21T01:03:08.296459+0000","last_clean":"2024-09-21T01:03:08.296459+0000","last_became_active":"2024-09-21T01:00:42.489630+0000","last_became_peered":"2024-09-21T01:00:42.489630+0000","last_unstale":"2024-09-21T01:03:08.296459+0000","last_undegraded":"2024-09-21T01:03:08.296459+0000","last_fullsized":"2024-09-21T01:03:08.296459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:02:15.811771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295660+0000","last_change":"2024-09-21T01:00:42.493041+0000","last_active":"2024-09-21T01:03:08.295660+0000","last_peered":"2024-09-21T01:03:08.295660+0000","last_clean":"2024-09-21T01:03:08.295660+0000","last_became_active":"2024-09-21T01:00:42.491898+0000","last_became_peered":"2024-09-21T01:00:42.491898+0000","last_unstale":"2024-09-21T01:03:08.295660+0000","last_undegraded":"2024-09-21T01:03:08.295660+0000","last_fullsized":"2024-09-21T01:03:08.295660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:10.031895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.291803+0000","last_change":"2024-09-21T01:00:42.499110+0000","last_active":"2024-09-21T01:03:08.291803+0000","last_peered":"2024-09-21T01:03:08.291803+0000","last_clean":"2024-09-21T01:03:08.291803+0000","last_became_active":"2024-09-21T01:00:42.498430+0000","last_became_peered":"2024-09-21T01:00:42.498430+0000","last_unstale":"2024-09-21T01:03:08.291803+0000","last_undegraded":"2024-09-21T01:03:08.291803+0000","last_fullsized":"2024-09-21T01:03:08.291803+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:31.565113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.295455+0000","last_change":"2024-09-21T01:01:10.124580+0000","last_active":"2024-09-21T01:03:08.295455+0000","last_peered":"2024-09-21T01:03:08.295455+0000","last_clean":"2024-09-21T01:03:08.295455+0000","last_became_active":"2024-09-21T01:01:10.124104+0000","last_became_peered":"2024-09-21T01:01:10.124104+0000","last_unstale":"2024-09-21T01:03:08.295455+0000","last_undegraded":"2024-09-21T01:03:08.295455+0000","last_fullsized":"2024-09-21T01:03:08.295455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:53:48.826443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.293894+0000","last_change":"2024-09-21T01:02:11.318295+0000","last_active":"2024-09-21T01:03:08.293894+0000","last_peered":"2024-09-21T01:03:08.293894+0000","last_clean":"2024-09-21T01:03:08.293894+0000","last_became_active":"2024-09-21T01:02:11.317912+0000","last_became_peered":"2024-09-21T01:02:11.317912+0000","last_unstale":"2024-09-21T01:03:08.293894+0000","last_undegraded":"2024-09-21T01:03:08.293894+0000","last_fullsized":"2024-09-21T01:03:08.293894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:53:47.063340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307810+0000","last_change":"2024-09-21T01:00:42.494979+0000","last_active":"2024-09-21T01:03:08.307810+0000","last_peered":"2024-09-21T01:03:08.307810+0000","last_clean":"2024-09-21T01:03:08.307810+0000","last_became_active":"2024-09-21T01:00:42.494520+0000","last_became_peered":"2024-09-21T01:00:42.494520+0000","last_unstale":"2024-09-21T01:03:08.307810+0000","last_undegraded":"2024-09-21T01:03:08.307810+0000","last_fullsized":"2024-09-21T01:03:08.307810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:22:36.882956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298761+0000","last_change":"2024-09-21T01:00:42.498085+0000","last_active":"2024-09-21T01:03:08.298761+0000","last_peered":"2024-09-21T01:03:08.298761+0000","last_clean":"2024-09-21T01:03:08.298761+0000","last_became_active":"2024-09-21T01:00:42.495013+0000","last_became_peered":"2024-09-21T01:00:42.495013+0000","last_unstale":"2024-09-21T01:03:08.298761+0000","last_undegraded":"2024-09-21T01:03:08.298761+0000","last_fullsized":"2024-09-21T01:03:08.298761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:06.859213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.296634+0000","last_change":"2024-09-21T01:00:42.494273+0000","last_active":"2024-09-21T01:03:08.296634+0000","last_peered":"2024-09-21T01:03:08.296634+0000","last_clean":"2024-09-21T01:03:08.296634+0000","last_became_active":"2024-09-21T01:00:42.490787+0000","last_became_peered":"2024-09-21T01:00:42.490787+0000","last_unstale":"2024-09-21T01:03:08.296634+0000","last_undegraded":"2024-09-21T01:03:08.296634+0000","last_fullsized":"2024-09-21T01:03:08.296634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:33:23.726705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290097+0000","last_change":"2024-09-21T01:00:42.502633+0000","last_active":"2024-09-21T01:03:08.290097+0000","last_peered":"2024-09-21T01:03:08.290097+0000","last_clean":"2024-09-21T01:03:08.290097+0000","last_became_active":"2024-09-21T01:00:42.502048+0000","last_became_peered":"2024-09-21T01:00:42.502048+0000","last_unstale":"2024-09-21T01:03:08.290097+0000","last_undegraded":"2024-09-21T01:03:08.290097+0000","last_fullsized":"2024-09-21T01:03:08.290097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:42:20.944178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.299853+0000","last_change":"2024-09-21T01:00:42.493862+0000","last_active":"2024-09-21T01:03:08.299853+0000","last_peered":"2024-09-21T01:03:08.299853+0000","last_clean":"2024-09-21T01:03:08.299853+0000","last_became_active":"2024-09-21T01:00:42.491061+0000","last_became_peered":"2024-09-21T01:00:42.491061+0000","last_unstale":"2024-09-21T01:03:08.299853+0000","last_undegraded":"2024-09-21T01:03:08.299853+0000","last_fullsized":"2024-09-21T01:03:08.299853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:08:59.461965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307223+0000","last_change":"2024-09-21T01:00:42.499144+0000","last_active":"2024-09-21T01:03:08.307223+0000","last_peered":"2024-09-21T01:03:08.307223+0000","last_clean":"2024-09-21T01:03:08.307223+0000","last_became_active":"2024-09-21T01:00:42.498600+0000","last_became_peered":"2024-09-21T01:00:42.498600+0000","last_unstale":"2024-09-21T01:03:08.307223+0000","last_undegraded":"2024-09-21T01:03:08.307223+0000","last_fullsized":"2024-09-21T01:03:08.307223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:30:19.411268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307652+0000","last_change":"2024-09-21T01:00:42.496316+0000","last_active":"2024-09-21T01:03:08.307652+0000","last_peered":"2024-09-21T01:03:08.307652+0000","last_clean":"2024-09-21T01:03:08.307652+0000","last_became_active":"2024-09-21T01:00:42.495965+0000","last_became_peered":"2024-09-21T01:00:42.495965+0000","last_unstale":"2024-09-21T01:03:08.307652+0000","last_undegraded":"2024-09-21T01:03:08.307652+0000","last_fullsized":"2024-09-21T01:03:08.307652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:23.905833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298544+0000","last_change":"2024-09-21T01:00:42.494948+0000","last_active":"2024-09-21T01:03:08.298544+0000","last_peered":"2024-09-21T01:03:08.298544+0000","last_clean":"2024-09-21T01:03:08.298544+0000","last_became_active":"2024-09-21T01:00:42.493198+0000","last_became_peered":"2024-09-21T01:00:42.493198+0000","last_unstale":"2024-09-21T01:03:08.298544+0000","last_undegraded":"2024-09-21T01:03:08.298544+0000","last_fullsized":"2024-09-21T01:03:08.298544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:16.707277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.304775+0000","last_change":"2024-09-21T01:00:42.492404+0000","last_active":"2024-09-21T01:03:08.304775+0000","last_peered":"2024-09-21T01:03:08.304775+0000","last_clean":"2024-09-21T01:03:08.304775+0000","last_became_active":"2024-09-21T01:00:42.491916+0000","last_became_peered":"2024-09-21T01:00:42.491916+0000","last_unstale":"2024-09-21T01:03:08.304775+0000","last_undegraded":"2024-09-21T01:03:08.304775+0000","last_fullsized":"2024-09-21T01:03:08.304775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:28:30.910269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.300375+0000","last_change":"2024-09-21T01:00:42.502439+0000","last_active":"2024-09-21T01:03:08.300375+0000","last_peered":"2024-09-21T01:03:08.300375+0000","last_clean":"2024-09-21T01:03:08.300375+0000","last_became_active":"2024-09-21T01:00:42.497595+0000","last_became_peered":"2024-09-21T01:00:42.497595+0000","last_unstale":"2024-09-21T01:03:08.300375+0000","last_undegraded":"2024-09-21T01:03:08.300375+0000","last_fullsized":"2024-09-21T01:03:08.300375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:26:19.690570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297218+0000","last_change":"2024-09-21T01:00:42.490922+0000","last_active":"2024-09-21T01:03:08.297218+0000","last_peered":"2024-09-21T01:03:08.297218+0000","last_clean":"2024-09-21T01:03:08.297218+0000","last_became_active":"2024-09-21T01:00:42.490112+0000","last_became_peered":"2024-09-21T01:00:42.490112+0000","last_unstale":"2024-09-21T01:03:08.297218+0000","last_undegraded":"2024-09-21T01:03:08.297218+0000","last_fullsized":"2024-09-21T01:03:08.297218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:54:10.240412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.302497+0000","last_change":"2024-09-21T01:00:42.500445+0000","last_active":"2024-09-21T01:03:08.302497+0000","last_peered":"2024-09-21T01:03:08.302497+0000","last_clean":"2024-09-21T01:03:08.302497+0000","last_became_active":"2024-09-21T01:00:42.499794+0000","last_became_peered":"2024-09-21T01:00:42.499794+0000","last_unstale":"2024-09-21T01:03:08.302497+0000","last_undegraded":"2024-09-21T01:03:08.302497+0000","last_fullsized":"2024-09-21T01:03:08.302497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:35.207448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.290792+0000","last_change":"2024-09-21T01:00:42.490400+0000","last_active":"2024-09-21T01:03:08.290792+0000","last_peered":"2024-09-21T01:03:08.290792+0000","last_clean":"2024-09-21T01:03:08.290792+0000","last_became_active":"2024-09-21T01:00:42.490038+0000","last_became_peered":"2024-09-21T01:00:42.490038+0000","last_unstale":"2024-09-21T01:03:08.290792+0000","last_undegraded":"2024-09-21T01:03:08.290792+0000","last_fullsized":"2024-09-21T01:03:08.290792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:42.797843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.306589+0000","last_change":"2024-09-21T01:00:42.490817+0000","last_active":"2024-09-21T01:03:08.306589+0000","last_peered":"2024-09-21T01:03:08.306589+0000","last_clean":"2024-09-21T01:03:08.306589+0000","last_became_active":"2024-09-21T01:00:42.490418+0000","last_became_peered":"2024-09-21T01:00:42.490418+0000","last_unstale":"2024-09-21T01:03:08.306589+0000","last_undegraded":"2024-09-21T01:03:08.306589+0000","last_fullsized":"2024-09-21T01:03:08.306589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:48.408471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.301598+0000","last_change":"2024-09-21T01:00:42.491836+0000","last_active":"2024-09-21T01:03:08.301598+0000","last_peered":"2024-09-21T01:03:08.301598+0000","last_clean":"2024-09-21T01:03:08.301598+0000","last_became_active":"2024-09-21T01:00:42.491251+0000","last_became_peered":"2024-09-21T01:00:42.491251+0000","last_unstale":"2024-09-21T01:03:08.301598+0000","last_undegraded":"2024-09-21T01:03:08.301598+0000","last_fullsized":"2024-09-21T01:03:08.301598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:27:11.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.294226+0000","last_change":"2024-09-21T01:02:11.138284+0000","last_active":"2024-09-21T01:03:08.294226+0000","last_peered":"2024-09-21T01:03:08.294226+0000","last_clean":"2024-09-21T01:03:08.294226+0000","last_became_active":"2024-09-21T01:02:11.137782+0000","last_became_peered":"2024-09-21T01:02:11.137782+0000","last_unstale":"2024-09-21T01:03:08.294226+0000","last_undegraded":"2024-09-21T01:03:08.294226+0000","last_fullsized":"2024-09-21T01:03:08.294226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:57.753015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.298298+0000","last_change":"2024-09-21T01:00:42.498250+0000","last_active":"2024-09-21T01:03:08.298298+0000","last_peered":"2024-09-21T01:03:08.298298+0000","last_clean":"2024-09-21T01:03:08.298298+0000","last_became_active":"2024-09-21T01:00:42.497908+0000","last_became_peered":"2024-09-21T01:00:42.497908+0000","last_unstale":"2024-09-21T01:03:08.298298+0000","last_undegraded":"2024-09-21T01:03:08.298298+0000","last_fullsized":"2024-09-21T01:03:08.298298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:39:30.430953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.292151+0000","last_change":"2024-09-21T01:00:42.505533+0000","last_active":"2024-09-21T01:03:08.292151+0000","last_peered":"2024-09-21T01:03:08.292151+0000","last_clean":"2024-09-21T01:03:08.292151+0000","last_became_active":"2024-09-21T01:00:42.505119+0000","last_became_peered":"2024-09-21T01:00:42.505119+0000","last_unstale":"2024-09-21T01:03:08.292151+0000","last_undegraded":"2024-09-21T01:03:08.292151+0000","last_fullsized":"2024-09-21T01:03:08.292151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:04:40.496375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.297492+0000","last_change":"2024-09-21T01:00:42.491188+0000","last_active":"2024-09-21T01:03:08.297492+0000","last_peered":"2024-09-21T01:03:08.297492+0000","last_clean":"2024-09-21T01:03:08.297492+0000","last_became_active":"2024-09-21T01:00:42.490577+0000","last_became_peered":"2024-09-21T01:00:42.490577+0000","last_unstale":"2024-09-21T01:03:08.297492+0000","last_undegraded":"2024-09-21T01:03:08.297492+0000","last_fullsized":"2024-09-21T01:03:08.297492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:37:22.272828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307974+0000","last_change":"2024-09-21T01:00:42.500532+0000","last_active":"2024-09-21T01:03:08.307974+0000","last_peered":"2024-09-21T01:03:08.307974+0000","last_clean":"2024-09-21T01:03:08.307974+0000","last_became_active":"2024-09-21T01:00:42.500180+0000","last_became_peered":"2024-09-21T01:00:42.500180+0000","last_unstale":"2024-09-21T01:03:08.307974+0000","last_undegraded":"2024-09-21T01:03:08.307974+0000","last_fullsized":"2024-09-21T01:03:08.307974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:17.170756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.303566+0000","last_change":"2024-09-21T01:00:42.490265+0000","last_active":"2024-09-21T01:03:08.303566+0000","last_peered":"2024-09-21T01:03:08.303566+0000","last_clean":"2024-09-21T01:03:08.303566+0000","last_became_active":"2024-09-21T01:00:42.489903+0000","last_became_peered":"2024-09-21T01:00:42.489903+0000","last_unstale":"2024-09-21T01:03:08.303566+0000","last_undegraded":"2024-09-21T01:03:08.303566+0000","last_fullsized":"2024-09-21T01:03:08.303566+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:40.814876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-21T01:03:08.307001+0000","last_change":"2024-09-21T01:00:42.498506+0000","last_active":"2024-09-21T01:03:08.307001+0000","last_peered":"2024-09-21T01:03:08.307001+0000","last_clean":"2024-09-21T01:03:08.307001+0000","last_became_active":"2024-09-21T01:00:42.498136+0000","last_became_peered":"2024-09-21T01:00:42.498136+0000","last_unstale":"2024-09-21T01:03:08.307001+0000","last_undegraded":"2024-09-21T01:03:08.307001+0000","last_fullsized":"2024-09-21T01:03:08.307001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:00:41.443926+0000","last_clean_scrub_stamp":"2024-09-21T01:00:41.443926+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:17:54.226231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":865644,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1144,"num_read_kb":785,"num_write":412,"num_write_kb":2231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2826240,"data_stored":2596932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678068,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":535286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874296,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29176,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93715976,"statfs":{"total":95995035648,"available":95965159424,"internally_reserved":0,"allocated":2142208,"data_stored":1594914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037823,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28444,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716708,"statfs":{"total":95995035648,"available":95965908992,"internally_reserved":0,"allocated":1454080,"data_stored":932812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234053,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28436,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716716,"statfs":{"total":95995035648,"available":95965917184,"internally_reserved":0,"allocated":1445888,"data_stored":928614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430282,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1650688,"data_stored":1132313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593808,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":530676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790039,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37408,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707744,"statfs":{"total":95995035648,"available":95956729856,"internally_reserved":0,"allocated":2052096,"data_stored":1516738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953565,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":536169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33040,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712112,"statfs":{"total":95995035648,"available":95961202688,"internally_reserved":0,"allocated":1839104,"data_stored":1317862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378727,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1138688,"data_stored":606135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542254,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1114112,"data_stored":604253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":532064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":73667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":806912,"data_stored":787276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":395784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":469864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:03:54.439 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:03:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:03:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:03:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:03:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:54 smithi150 ceph-mon[34945]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:03:54.861 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:55.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.520+0000 7f21af600640 1 -- 172.21.15.39:0/3692190586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8078290 msgr2=0x7f21a8114200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:55.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.520+0000 7f21af600640 1 --2- 172.21.15.39:0/3692190586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8078290 0x7f21a8114200 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2194009900 tx=0x7f219402a410 comp rx=0 tx=0).stop 2024-09-21T01:03:55.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 -- 172.21.15.39:0/3692190586 shutdown_connections 2024-09-21T01:03:55.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 --2- 172.21.15.39:0/3692190586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21a8114740 0x7f21a8116b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 --2- 172.21.15.39:0/3692190586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8078290 0x7f21a8114200 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 --2- 172.21.15.39:0/3692190586 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21a8077940 0x7f21a8077d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 -- 172.21.15.39:0/3692190586 >> 172.21.15.39:0/3692190586 conn(0x7f21a8101810 msgr2=0x7f21a8103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:55.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 -- 172.21.15.39:0/3692190586 shutdown_connections 2024-09-21T01:03:55.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.521+0000 7f21af600640 1 -- 172.21.15.39:0/3692190586 wait complete. 2024-09-21T01:03:55.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.522+0000 7f21af600640 1 Processor -- start 2024-09-21T01:03:55.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.522+0000 7f21af600640 1 -- start start 2024-09-21T01:03:55.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21af600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21a8077940 0x7f21a81a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:55.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21af600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21a8078290 0x7f21a81a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:55.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21af600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 0x7f21a81a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21af600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f21a811a070 con 0x7f21a8114740 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21af600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f21a8119ef0 con 0x7f21a8077940 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21af600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f21a811a1f0 con 0x7f21a8078290 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21ad600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 0x7f21a81a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21ad600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 0x7f21a81a6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58524/0 (socket says 172.21.15.39:58524) 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21a8078290 0x7f21a81a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21acc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21a8077940 0x7f21a81a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:55.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.523+0000 7f21ad600640 1 -- 172.21.15.39:0/60859674 learned_addr learned my addr 172.21.15.39:0/60859674 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:55.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21ad600640 1 -- 172.21.15.39:0/60859674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21a8078290 msgr2=0x7f21a81a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:55.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21ad600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21a8078290 0x7f21a81a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21ad600640 1 -- 172.21.15.39:0/60859674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21a8077940 msgr2=0x7f21a81a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:55.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21ad600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21a8077940 0x7f21a81a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21ad600640 1 -- 172.21.15.39:0/60859674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a81a6a90 con 0x7f21a8114740 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21ad600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 0x7f21a81a6360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f21a000eb10 tx=0x7f21a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0004b20 con 0x7f21a8114740 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a81a6d80 con 0x7f21a8114740 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.524+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21a0004cc0 con 0x7f21a8114740 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.525+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a81ad4e0 con 0x7f21a8114740 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.525+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0018630 con 0x7f21a8114740 2024-09-21T01:03:55.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.526+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a8077d50 con 0x7f21a8114740 2024-09-21T01:03:55.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.528+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f21a0005000 con 0x7f21a8114740 2024-09-21T01:03:55.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.529+0000 7f21a5600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21900b65f0 0x7f21900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:55.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.529+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f21a00e7fe0 con 0x7f21a8114740 2024-09-21T01:03:55.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.530+0000 7f21acc00640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21900b65f0 0x7f21900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:55.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.530+0000 7f21acc00640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21900b65f0 0x7f21900b8b00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f219c00bbe0 tx=0x7f219c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:55.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.533+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f21a00ae270 con 0x7f21a8114740 2024-09-21T01:03:55.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:55 smithi150 ceph-mon[34945]: pgmap v401: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 52 op/s 2024-09-21T01:03:55.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:55 smithi160 ceph-mon[34696]: pgmap v401: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 52 op/s 2024-09-21T01:03:55.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:55 smithi039 ceph-mon[30415]: pgmap v401: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 175 KiB/s wr, 52 op/s 2024-09-21T01:03:55.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.773+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21a81a1c50 con 0x7f21a8114740 2024-09-21T01:03:55.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.774+0000 7f21a5600640 1 -- 172.21.15.39:0/60859674 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7f21a00b2650 con 0x7f21a8114740 2024-09-21T01:03:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:55.776 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:03:55.776 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.776+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21900b65f0 msgr2=0x7f21900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.776+0000 7f21af600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21900b65f0 0x7f21900b8b00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f219c00bbe0 tx=0x7f219c00b390 comp rx=0 tx=0).stop 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.776+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 msgr2=0x7f21a81a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.776+0000 7f21af600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 0x7f21a81a6360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f21a000eb10 tx=0x7f21a000c2d0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 shutdown_connections 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8114740 0x7f21a81a6360 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f21a8078290 0x7f21a81a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f21900b65f0 0x7f21900b8b00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 --2- 172.21.15.39:0/60859674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21a8077940 0x7f21a81a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 >> 172.21.15.39:0/60859674 conn(0x7f21a8101810 msgr2=0x7f21a81167e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 shutdown_connections 2024-09-21T01:03:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:55.777+0000 7f21af600640 1 -- 172.21.15.39:0/60859674 wait complete. 2024-09-21T01:03:56.538 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/60859674' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:56.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/60859674' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:56.577 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:03:56.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/60859674' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:56.960 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:57.353 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:57 smithi039 ceph-mon[30415]: pgmap v402: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:57 smithi150 ceph-mon[34945]: pgmap v402: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:57.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:57 smithi160 ceph-mon[34696]: pgmap v402: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 -- 172.21.15.39:0/3078306467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc81094a0 msgr2=0x7f5bc81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 --2- 172.21.15.39:0/3078306467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc81094a0 0x7f5bc81098b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009900 tx=0x7f5bb002a410 comp rx=0 tx=0).stop 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 -- 172.21.15.39:0/3078306467 shutdown_connections 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 --2- 172.21.15.39:0/3078306467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bc8109df0 0x7f5bc8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 --2- 172.21.15.39:0/3078306467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc81094a0 0x7f5bc81098b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 --2- 172.21.15.39:0/3078306467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 -- 172.21.15.39:0/3078306467 >> 172.21.15.39:0/3078306467 conn(0x7f5bc807b850 msgr2=0x7f5bc807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:57.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 -- 172.21.15.39:0/3078306467 shutdown_connections 2024-09-21T01:03:57.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.677+0000 7f5bcd200640 1 -- 172.21.15.39:0/3078306467 wait complete. 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.678+0000 7f5bcd200640 1 Processor -- start 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 -- start start 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bc81094a0 0x7f5bc819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8109df0 0x7f5bc81a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc8113ab0 con 0x7f5bc8109df0 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc8113930 con 0x7f5bc81094a0 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bcd200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc8113c30 con 0x7f5bc8103b60 2024-09-21T01:03:57.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bc5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bc6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8109df0 0x7f5bc81a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bc5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc819c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39368/0 (socket says 172.21.15.39:39368) 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bc5e00640 1 -- 172.21.15.39:0/1648723789 learned_addr learned my addr 172.21.15.39:0/1648723789 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.679+0000 7f5bbfe00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bc81094a0 0x7f5bc819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bc5e00640 1 -- 172.21.15.39:0/1648723789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bc81094a0 msgr2=0x7f5bc819cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bc5e00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bc81094a0 0x7f5bc819cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bc5e00640 1 -- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8109df0 msgr2=0x7f5bc81a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bc5e00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8109df0 0x7f5bc81a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bc5e00640 1 -- 172.21.15.39:0/1648723789 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc81a2630 con 0x7f5bc8103b60 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bc5e00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc819c6c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800d950 tx=0x7f5bb800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8017030 con 0x7f5bc8103b60 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc81a2920 con 0x7f5bc8103b60 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.680+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bb8004020 con 0x7f5bc8103b60 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.681+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb80047a0 con 0x7f5bc8103b60 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.681+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc81a9080 con 0x7f5bc8103b60 2024-09-21T01:03:57.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.682+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc8103f70 con 0x7f5bc8103b60 2024-09-21T01:03:57.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.684+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f5bb8009a30 con 0x7f5bc8103b60 2024-09-21T01:03:57.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.686+0000 7f5bbea00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bac0b65f0 0x7f5bac0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:57.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.686+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5bb80e7aa0 con 0x7f5bc8103b60 2024-09-21T01:03:57.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.686+0000 7f5bbfe00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bac0b65f0 0x7f5bac0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:57.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.687+0000 7f5bbfe00640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bac0b65f0 0x7f5bac0b8b00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009230 tx=0x7f5bb0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:57.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.688+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5bb80adcd0 con 0x7f5bc8103b60 2024-09-21T01:03:57.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.927+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bc819d7f0 con 0x7f5bc8103b60 2024-09-21T01:03:57.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.928+0000 7f5bbea00640 1 -- 172.21.15.39:0/1648723789 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7f5bc819d7f0 con 0x7f5bc8103b60 2024-09-21T01:03:57.930 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:57.930 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:03:57.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bac0b65f0 msgr2=0x7f5bac0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:57.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bac0b65f0 0x7f5bac0b8b00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009230 tx=0x7f5bb0031400 comp rx=0 tx=0).stop 2024-09-21T01:03:57.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 msgr2=0x7f5bc819c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:57.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc819c6c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800d950 tx=0x7f5bb800de10 comp rx=0 tx=0).stop 2024-09-21T01:03:57.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 shutdown_connections 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8109df0 0x7f5bc81a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bac0b65f0 0x7f5bac0b8b00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bc81094a0 0x7f5bc819cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 --2- 172.21.15.39:0/1648723789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5bc8103b60 0x7f5bc819c6c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.931+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 >> 172.21.15.39:0/1648723789 conn(0x7f5bc807b850 msgr2=0x7f5bc81073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.932+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 shutdown_connections 2024-09-21T01:03:57.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:57.932+0000 7f5bcd200640 1 -- 172.21.15.39:0/1648723789 wait complete. 2024-09-21T01:03:58.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:58 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1648723789' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:58.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:58 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1648723789' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:58.751 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:03:58.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:58 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1648723789' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:03:59.130 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:03:59.600 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:03:59 smithi039 ceph-mon[30415]: pgmap v403: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 174 KiB/s wr, 53 op/s 2024-09-21T01:03:59.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:03:59 smithi160 ceph-mon[34696]: pgmap v403: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 174 KiB/s wr, 53 op/s 2024-09-21T01:03:59.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 -- 172.21.15.39:0/689513260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 msgr2=0x7ff0000761e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:59.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 --2- 172.21.15.39:0/689513260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0000761e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009860 tx=0x7fefe802a340 comp rx=0 tx=0).stop 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 -- 172.21.15.39:0/689513260 shutdown_connections 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 --2- 172.21.15.39:0/689513260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000114280 0x7ff000116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 --2- 172.21.15.39:0/689513260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0000761e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 --2- 172.21.15.39:0/689513260 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff000077380 0x7ff0000757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 -- 172.21.15.39:0/689513260 >> 172.21.15.39:0/689513260 conn(0x7ff0000ff440 msgr2=0x7ff000101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 -- 172.21.15.39:0/689513260 shutdown_connections 2024-09-21T01:03:59.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.778+0000 7ff004a00640 1 -- 172.21.15.39:0/689513260 wait complete. 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.779+0000 7ff004a00640 1 Processor -- start 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.779+0000 7ff004a00640 1 -- start start 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.779+0000 7ff004a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0001a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.779+0000 7ff004a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 0x7ff0001a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7ff004a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff000114280 0x7ff0001a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7ff004a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff000119ec0 con 0x7ff000077380 2024-09-21T01:03:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7ff004a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff000119d40 con 0x7ff000075dd0 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7ff004a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff00011a040 con 0x7ff000114280 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 0x7ff0001a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7fefff400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0001a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 0x7ff0001a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58562/0 (socket says 172.21.15.39:58562) 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7fefff400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0001a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:51786/0 (socket says 172.21.15.39:51786) 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 -- 172.21.15.39:0/3679137178 learned_addr learned my addr 172.21.15.39:0/3679137178 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:03:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7fefffe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff000114280 0x7ff0001a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 -- 172.21.15.39:0/3679137178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff000114280 msgr2=0x7ff0001a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff000114280 0x7ff0001a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 -- 172.21.15.39:0/3679137178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 msgr2=0x7ff0001a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0001a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.780+0000 7feff6a00640 1 -- 172.21.15.39:0/3679137178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0001a6860 con 0x7ff000077380 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.781+0000 7feff6a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 0x7ff0001a0e90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fefe8000c00 tx=0x7fefe8032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.781+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8040020 con 0x7ff000077380 2024-09-21T01:03:59.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.781+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0001a6af0 con 0x7ff000077380 2024-09-21T01:03:59.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.781+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefe8030e10 con 0x7ff000077380 2024-09-21T01:03:59.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.781+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0001ad2b0 con 0x7ff000077380 2024-09-21T01:03:59.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.781+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8037040 con 0x7ff000077380 2024-09-21T01:03:59.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.783+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0000761e0 con 0x7ff000077380 2024-09-21T01:03:59.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.785+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fefe8043020 con 0x7ff000077380 2024-09-21T01:03:59.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.786+0000 7feffcc00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fefe40b65f0 0x7fefe40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:03:59.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.786+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fefe81075d0 con 0x7ff000077380 2024-09-21T01:03:59.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.786+0000 7fefff400640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fefe40b65f0 0x7fefe40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:03:59.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.787+0000 7fefff400640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fefe40b65f0 0x7fefe40b8b00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feff0005fd0 tx=0x7feff0009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:03:59.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.790+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fefe804a030 con 0x7ff000077380 2024-09-21T01:03:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:03:59 smithi150 ceph-mon[34945]: pgmap v403: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 174 KiB/s wr, 53 op/s 2024-09-21T01:03:59.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.993+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff000112fe0 con 0x7ff000077380 2024-09-21T01:03:59.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.995+0000 7feffcc00640 1 -- 172.21.15.39:0/3679137178 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25331 (secure 0 0 0) 0x7fefe80cd800 con 0x7ff000077380 2024-09-21T01:03:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:03:59.997 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":90,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:02:10.480352+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fefe40b65f0 msgr2=0x7fefe40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fefe40b65f0 0x7fefe40b8b00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feff0005fd0 tx=0x7feff0009500 comp rx=0 tx=0).stop 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 msgr2=0x7ff0001a0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 0x7ff0001a0e90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fefe8000c00 tx=0x7fefe8032d40 comp rx=0 tx=0).stop 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 shutdown_connections 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff000114280 0x7ff0001a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff000077380 0x7ff0001a0e90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fefe40b65f0 0x7fefe40b8b00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.997+0000 7ff004a00640 1 --2- 172.21.15.39:0/3679137178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff000075dd0 0x7ff0001a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.998+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 >> 172.21.15.39:0/3679137178 conn(0x7ff0000ff440 msgr2=0x7ff000110180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:03:59.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.998+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 shutdown_connections 2024-09-21T01:04:00.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:03:59.998+0000 7ff004a00640 1 -- 172.21.15.39:0/3679137178 wait complete. 2024-09-21T01:04:00.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:00 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3679137178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:04:00.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:00 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3679137178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:04:00.654 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:04:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:04:00.319+0000 7f4d89c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:04:00.654 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:04:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:04:00.319+0000 7f4d89c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:04:00.654 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:04:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:04:00.319+0000 7f4d89c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:04:00.654 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:04:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:04:00.319+0000 7f4d89c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:04:00.654 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:04:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:04:00.319+0000 7f4d89c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T01:04:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:00 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3679137178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:04:00.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mon.a --format=json config get run_dir 2024-09-21T01:04:01.198 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:01.483 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:01 smithi039 ceph-mon[30415]: pgmap v404: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-21T01:04:01.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:01 smithi160 ceph-mon[34696]: pgmap v404: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-21T01:04:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:01 smithi150 ceph-mon[34945]: pgmap v404: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-21T01:04:02.761 INFO:teuthology.orchestra.run.smithi039.stdout:{"run_dir":"/var/run/ceph"} 2024-09-21T01:04:02.761 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-21T01:04:02.761 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 7864 2024-09-21T01:04:02.761 DEBUG:tasks.quiescer:Yielding 2024-09-21T01:04:02.762 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T01:04:02.777 INFO:tasks.workunit:Pulling workunits from ref 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:04:02.777 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T01:04:02.777 INFO:tasks.workunit:timeout=3h 2024-09-21T01:04:02.777 INFO:tasks.workunit:cleanup=True 2024-09-21T01:04:02.777 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T01:04:02.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:04:02.818 INFO:teuthology.orchestra.run.smithi039.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-21T01:04:02.818 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-21T01:04:02.818 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-21T01:04:02.818 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:04:02.819 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-21T01:04:02.819 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 01:01:53.112726278 +0000 2024-09-21T01:04:02.819 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 01:01:53.112726278 +0000 2024-09-21T01:04:02.819 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 01:03:43.243790986 +0000 2024-09-21T01:04:02.819 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-21 01:01:53.112726278 +0000 2024-09-21T01:04:02.819 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-21T01:04:02.819 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-21T01:04:02.931 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:04:02.970 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T01:04:03.207 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:03.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:03 smithi160 ceph-mon[34696]: pgmap v405: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-09-21T01:04:03.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:03 smithi160 ceph-mon[34696]: mgrmap e24: x(active, since 10m), standbys: z, y 2024-09-21T01:04:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:03 smithi039 ceph-mon[30415]: pgmap v405: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-09-21T01:04:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:03 smithi039 ceph-mon[30415]: mgrmap e24: x(active, since 10m), standbys: z, y 2024-09-21T01:04:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:03 smithi150 ceph-mon[34945]: pgmap v405: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-09-21T01:04:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:03 smithi150 ceph-mon[34945]: mgrmap e24: x(active, since 10m), standbys: z, y 2024-09-21T01:04:03.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 -- 172.21.15.39:0/3332924828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 msgr2=0x7f1174110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:03.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 --2- 172.21.15.39:0/3332924828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 0x7f1174110320 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f116c009900 tx=0x7f116c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:04:03.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 -- 172.21.15.39:0/3332924828 shutdown_connections 2024-09-21T01:04:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 --2- 172.21.15.39:0/3332924828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 0x7f1174110320 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 --2- 172.21.15.39:0/3332924828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174105320 0x7f117410d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 --2- 172.21.15.39:0/3332924828 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f11741049d0 0x7f1174104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 -- 172.21.15.39:0/3332924828 >> 172.21.15.39:0/3332924828 conn(0x7f1174079740 msgr2=0x7f1174079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 -- 172.21.15.39:0/3332924828 shutdown_connections 2024-09-21T01:04:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.898+0000 7f117a600640 1 -- 172.21.15.39:0/3332924828 wait complete. 2024-09-21T01:04:03.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.899+0000 7f117a600640 1 Processor -- start 2024-09-21T01:04:03.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 -- start start 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11741049d0 0x7f1174102710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 0x7f1174102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 0x7f1174103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1174113b00 con 0x7f117410de30 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1174113980 con 0x7f11741049d0 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f117a600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1174113c80 con 0x7f1174105320 2024-09-21T01:04:03.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f116aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 0x7f1174102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f1173e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 0x7f1174103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f1173400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11741049d0 0x7f1174102710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f116aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 0x7f1174102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52414/0 (socket says 172.21.15.39:52414) 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.900+0000 7f116aa00640 1 -- 172.21.15.39:0/3981115502 learned_addr learned my addr 172.21.15.39:0/3981115502 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.901+0000 7f116aa00640 1 -- 172.21.15.39:0/3981115502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11741049d0 msgr2=0x7f1174102710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.901+0000 7f116aa00640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11741049d0 0x7f1174102710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:03.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.901+0000 7f116aa00640 1 -- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 msgr2=0x7f1174103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.901+0000 7f116aa00640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 0x7f1174103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.901+0000 7f116aa00640 1 -- 172.21.15.39:0/3981115502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11741a87f0 con 0x7f1174105320 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.901+0000 7f116aa00640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 0x7f1174102c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f115c00e9f0 tx=0x7f115c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.902+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c00ccc0 con 0x7f1174105320 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.902+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11741a8ae0 con 0x7f1174105320 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.902+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f115c0108e0 con 0x7f1174105320 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.902+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c018860 con 0x7f1174105320 2024-09-21T01:04:03.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.902+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741a9070 con 0x7f1174105320 2024-09-21T01:04:03.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.903+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f117406c240 con 0x7f1174105320 2024-09-21T01:04:03.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.906+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f115c010430 con 0x7f1174105320 2024-09-21T01:04:03.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.907+0000 7f1170c00640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f11580b6680 0x7f11580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:03.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.908+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f115c0e81c0 con 0x7f1174105320 2024-09-21T01:04:03.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.908+0000 7f1173400640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f11580b6680 0x7f11580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:03.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.909+0000 7f1173400640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f11580b6680 0x7f11580b8b90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1174104160 tx=0x7f1164009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:03.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:03.909+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f115c0b2710 con 0x7f1174105320 2024-09-21T01:04:04.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.143+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1174104de0 con 0x7f1174105320 2024-09-21T01:04:04.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.144+0000 7f1170c00640 1 -- 172.21.15.39:0/3981115502 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7f115c0b7650 con 0x7f1174105320 2024-09-21T01:04:04.146 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:04:04.146 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:04:04.147 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f11580b6680 msgr2=0x7f11580b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f11580b6680 0x7f11580b8b90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1174104160 tx=0x7f1164009500 comp rx=0 tx=0).stop 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 msgr2=0x7f1174102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 0x7f1174102c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f115c00e9f0 tx=0x7f115c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 shutdown_connections 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f117410de30 0x7f1174103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1174105320 0x7f1174102c50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f11580b6680 0x7f11580b8b90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 --2- 172.21.15.39:0/3981115502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11741049d0 0x7f1174102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.147+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 >> 172.21.15.39:0/3981115502 conn(0x7f1174079740 msgr2=0x7f11740ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.148+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 shutdown_connections 2024-09-21T01:04:04.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:04.148+0000 7f117a600640 1 -- 172.21.15.39:0/3981115502 wait complete. 2024-09-21T01:04:04.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:04 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3981115502' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:04.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:04 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3981115502' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:04.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:04 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3981115502' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:04.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T01:00:43.758917+0000', 'modified': '2024-09-21T01:03:44.185282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24764, 'mds_1': 24760, 'mds_2': 24742}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24715': {'gid': 24715, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6835/2680876155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6834', 'nonce': 2680876155}, {'type': 'v1', 'addr': '172.21.15.160:6835', 'nonce': 2680876155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24725': {'gid': 24725, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/555411539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 555411539}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 555411539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24727': {'gid': 24727, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.160:6837/1965195431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6836', 'nonce': 1965195431}, {'type': 'v1', 'addr': '172.21.15.160:6837', 'nonce': 1965195431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24742': {'gid': 24742, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.160:6839/2403371273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.160:6838', 'nonce': 2403371273}, {'type': 'v1', 'addr': '172.21.15.160:6839', 'nonce': 2403371273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24760': {'gid': 24760, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.39:6841/3868264331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6840', 'nonce': 3868264331}, {'type': 'v1', 'addr': '172.21.15.39:6841', 'nonce': 3868264331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24764': {'gid': 24764, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.150:6839/103174700', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 103174700}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 103174700}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24764, 'qdb_cluster': [24742, 24760, 24764]} 2024-09-21T01:04:04.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-21T01:04:04.958 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T01:04:05.381 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:05.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:05 smithi039 ceph-mon[30415]: pgmap v406: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-09-21T01:04:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:05 smithi150 ceph-mon[34945]: pgmap v406: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-09-21T01:04:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:05 smithi160 ceph-mon[34696]: pgmap v406: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-09-21T01:04:06.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.037+0000 7fea8b200640 1 -- 172.21.15.39:0/1702781087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 msgr2=0x7fea8c10d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.037+0000 7fea8b200640 1 --2- 172.21.15.39:0/1702781087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c10d670 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fea78009900 tx=0x7fea7802a410 comp rx=0 tx=0).stop 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 -- 172.21.15.39:0/1702781087 shutdown_connections 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 --2- 172.21.15.39:0/1702781087 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c10dc30 0x7fea8c110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 --2- 172.21.15.39:0/1702781087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c10d670 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 --2- 172.21.15.39:0/1702781087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea8c1047d0 0x7fea8c104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 -- 172.21.15.39:0/1702781087 >> 172.21.15.39:0/1702781087 conn(0x7fea8c0fe2d0 msgr2=0x7fea8c100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:06.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 -- 172.21.15.39:0/1702781087 shutdown_connections 2024-09-21T01:04:06.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.038+0000 7fea8b200640 1 -- 172.21.15.39:0/1702781087 wait complete. 2024-09-21T01:04:06.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.039+0000 7fea8b200640 1 Processor -- start 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.039+0000 7fea8b200640 1 -- start start 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.039+0000 7fea8b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c1047d0 0x7fea8c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.039+0000 7fea8b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea8b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea8c10dc30 0x7fea8c19fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea8b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c1138e0 con 0x7fea8c105120 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea8b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c113760 con 0x7fea8c10dc30 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea8b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c113a60 con 0x7fea8c1047d0 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49076/0 (socket says 172.21.15.39:49076) 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89400640 1 -- 172.21.15.39:0/3443335279 learned_addr learned my addr 172.21.15.39:0/3443335279 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89e00640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c1047d0 0x7fea8c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea8a800640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea8c10dc30 0x7fea8c19fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89e00640 1 -- 172.21.15.39:0/3443335279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea8c10dc30 msgr2=0x7fea8c19fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89e00640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea8c10dc30 0x7fea8c19fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89e00640 1 -- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 msgr2=0x7fea8c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89e00640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.040+0000 7fea89e00640 1 -- 172.21.15.39:0/3443335279 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c1a01e0 con 0x7fea8c1047d0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.041+0000 7fea89e00640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c1047d0 0x7fea8c19a270 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fea7c005fd0 tx=0x7fea7c00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.041+0000 7fea82a00640 1 -- 172.21.15.39:0/3443335279 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7c009c30 con 0x7fea8c1047d0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.041+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8c1a04d0 con 0x7fea8c1047d0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.041+0000 7fea82a00640 1 -- 172.21.15.39:0/3443335279 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea7c009dd0 con 0x7fea8c1047d0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.041+0000 7fea82a00640 1 -- 172.21.15.39:0/3443335279 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7c003e20 con 0x7fea8c1047d0 2024-09-21T01:04:06.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.041+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8c1a6c30 con 0x7fea8c1047d0 2024-09-21T01:04:06.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.044+0000 7fea97a7b740 1 Processor -- start 2024-09-21T01:04:06.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.044+0000 7fea97a7b740 1 -- start start 2024-09-21T01:04:06.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 0x55c103a17670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1036480a0 0x55c103a17bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea97a7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c10398ed50 con 0x55c1036480a0 2024-09-21T01:04:06.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea97a7b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c10397e180 con 0x55c103670f60 2024-09-21T01:04:06.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea97a7b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c103981f90 con 0x55c103650280 2024-09-21T01:04:06.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea8a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 0x55c103a17670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea8a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 0x55c103a17670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39336/0 (socket says 172.21.15.39:39336) 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea8a800640 1 -- 172.21.15.39:0/3170580083 learned_addr learned my addr 172.21.15.39:0/3170580083 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.045+0000 7fea89400640 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea8a800640 1 -- 172.21.15.39:0/3170580083 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 msgr2=0x55c103a180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea8a800640 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a180f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea8a800640 1 -- 172.21.15.39:0/3170580083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1036480a0 msgr2=0x55c103a17bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea8a800640 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1036480a0 0x55c103a17bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea8a800640 1 -- 172.21.15.39:0/3170580083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c103948a70 con 0x55c103670f60 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea82a00640 1 -- 172.21.15.39:0/3443335279 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fea7c004000 con 0x7fea8c1047d0 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea89400640 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea8a800640 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 0x55c103a17670 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fea84009f50 tx=0x7fea840118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe01fcfdf10cf21b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea77400640 1 -- 172.21.15.39:0/3170580083 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84004d20 con 0x55c103670f60 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea77400640 1 -- 172.21.15.39:0/3170580083 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea84004050 con 0x55c103670f60 2024-09-21T01:04:06.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.046+0000 7fea77400640 1 -- 172.21.15.39:0/3170580083 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84005280 con 0x55c103670f60 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 -- 172.21.15.39:0/3170580083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 msgr2=0x55c103a17670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 0x55c103a17670 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fea84009f50 tx=0x7fea840118d0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea82a00640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea700b6700 0x7fea700b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea82a00640 1 -- 172.21.15.39:0/3443335279 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea7c0e7600 con 0x7fea8c1047d0 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 -- 172.21.15.39:0/3170580083 shutdown_connections 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1036480a0 0x55c103a17bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3170580083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c103670f60 0x55c103a17670 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea97a7b740 1 -- 172.21.15.39:0/3170580083 >> 172.21.15.39:0/3170580083 conn(0x55c103665350 msgr2=0x55c103a15560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.047+0000 7fea8a800640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea700b6700 0x7fea700b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.048+0000 7fea97a7b740 1 -- 172.21.15.39:0/3170580083 shutdown_connections 2024-09-21T01:04:06.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.048+0000 7fea97a7b740 1 -- 172.21.15.39:0/3170580083 wait complete. 2024-09-21T01:04:06.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.048+0000 7fea97a7b740 1 -- start start 2024-09-21T01:04:06.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.048+0000 7fea8a800640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea700b6700 0x7fea700b8c10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fea84005fd0 tx=0x7fea84004960 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:06.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.048+0000 7fea97a7b740 1 Processor -- start 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a27b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a334b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c1037df5f0 con 0x55c103670f60 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c103919460 con 0x55c1036480a0 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c103a2a180 con 0x55c103650280 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea97a7b740 10 client.0 _ll_register_callbacks cb 0x55c103929210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:06.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea89400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a27b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea8a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea89e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea89400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a27b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52446/0 (socket says 172.21.15.39:52446) 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea89400640 1 -- 172.21.15.39:0/622083918 learned_addr learned my addr 172.21.15.39:0/622083918 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.049+0000 7fea89e00640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a275d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39346/0 (socket says 172.21.15.39:39346) 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea89400640 1 -- 172.21.15.39:0/622083918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 msgr2=0x55c103a275d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea89400640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a275d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea89400640 1 -- 172.21.15.39:0/622083918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 msgr2=0x55c103a334b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea89400640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a334b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea89400640 1 -- 172.21.15.39:0/622083918 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1036dc540 con 0x55c103650280 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea8a800640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:06.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea89400640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a27b10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fea7802aa00 tx=0x7fea78004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9b0ef9a37c41dd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.050+0000 7fea6be00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea6be00640 1 -- 172.21.15.39:0/622083918 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78035040 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c10391ba70 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea6be00640 1 -- 172.21.15.39:0/622083918 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea78005220 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea6be00640 1 -- 172.21.15.39:0/622083918 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea78005420 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea97a7b740 10 client.25462 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c1039366f0 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.051+0000 7fea6be00640 1 -- 172.21.15.39:0/622083918 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c10391ba70 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea6be00640 1 -- 172.21.15.39:0/622083918 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fea780038b0 con 0x55c103650280 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea97a7b740 10 client.25462 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea97a7b740 10 client.25462 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea97a7b740 20 client.25462 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea97a7b740 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c10362f8e0 0x55c103a399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea97a7b740 4 client.25462 mds_command: new command op to 24742 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:04:06.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 --> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c103990ca0 con 0x55c10362f8e0 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.052+0000 7fea89e00640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c10362f8e0 0x55c103a399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.053+0000 7fea89e00640 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c10362f8e0 0x55c103a399c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d976f34cbaf06cc server_cookie=ef96ca076afc0490 in_seq=0 out_seq=0 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.053+0000 7fea6be00640 10 client.25462 ms_handle_connect on v2:172.21.15.160:6838/2403371273 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea6be00640 1 -- 172.21.15.39:0/622083918 <== mds.2 v2:172.21.15.160:6838/2403371273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c103990ca0 con 0x55c10362f8e0 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea6be00640 10 client.25462 handle_command_reply: tid=0 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 client.25462 shutdown 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 msgr2=0x55c103a27b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a27b10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fea7802aa00 tx=0x7fea78004430 comp rx=0 tx=0).stop 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 shutdown_connections 2024-09-21T01:04:06.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c10362f8e0 0x55c103a399c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a27b10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 --2- 172.21.15.39:0/622083918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea6be00640 0 client.25462 ms_handle_reset on v2:172.21.15.160:6838/2403371273 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 >> 172.21.15.39:0/622083918 conn(0x55c103665350 msgr2=0x55c103a1a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 shutdown_connections 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 1 -- 172.21.15.39:0/622083918 wait complete. 2024-09-21T01:04:06.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.054+0000 7fea97a7b740 20 client.25462 trim_cache size 0 max 16384 2024-09-21T01:04:06.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.073+0000 7fea97a7b740 1 Processor -- start 2024-09-21T01:04:06.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.073+0000 7fea97a7b740 1 -- start start 2024-09-21T01:04:06.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.073+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.073+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 0x55c103a275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.073+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103670f60 0x55c103a27b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea97a7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c103a37d00 con 0x55c103650280 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea97a7b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c103a383d0 con 0x55c1036480a0 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea97a7b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c103a38aa0 con 0x55c103670f60 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 0x55c103a275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 0x55c103a275d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49084/0 (socket says 172.21.15.39:49084) 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 -- 172.21.15.39:0/3514342549 learned_addr learned my addr 172.21.15.39:0/3514342549 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89e00640 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea8a800640 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103670f60 0x55c103a27b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 -- 172.21.15.39:0/3514342549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103670f60 msgr2=0x55c103a27b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103670f60 0x55c103a27b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 -- 172.21.15.39:0/3514342549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 msgr2=0x55c103a399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a399c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.074+0000 7fea89400640 1 -- 172.21.15.39:0/3514342549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c103a18450 con 0x55c103650280 2024-09-21T01:04:06.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.075+0000 7fea89400640 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 0x55c103a275d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea78001eb0 tx=0x7fea7802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1f7cd5c8017ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.075+0000 7fea76a00640 1 -- 172.21.15.39:0/3514342549 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78056070 con 0x55c103650280 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.075+0000 7fea76a00640 1 -- 172.21.15.39:0/3514342549 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea780044a0 con 0x55c103650280 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.075+0000 7fea76a00640 1 -- 172.21.15.39:0/3514342549 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78035070 con 0x55c103650280 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 -- 172.21.15.39:0/3514342549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 msgr2=0x55c103a275d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 0x55c103a275d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea78001eb0 tx=0x7fea7802a410 comp rx=0 tx=0).stop 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 -- 172.21.15.39:0/3514342549 shutdown_connections 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103670f60 0x55c103a27b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103650280 0x55c103a275d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3514342549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 -- 172.21.15.39:0/3514342549 >> 172.21.15.39:0/3514342549 conn(0x55c103665350 msgr2=0x55c103a37410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 -- 172.21.15.39:0/3514342549 shutdown_connections 2024-09-21T01:04:06.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 -- 172.21.15.39:0/3514342549 wait complete. 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.076+0000 7fea97a7b740 1 -- start start 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 Processor -- start 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a3c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a3c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a411f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c103a39840 con 0x55c103670f60 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c103a39170 con 0x55c1036480a0 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c103a39f10 con 0x55c103650280 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea97a7b740 10 client.0 _ll_register_callbacks cb 0x7fea78004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea89e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a3c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea8a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.077+0000 7fea89e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a3c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39366/0 (socket says 172.21.15.39:39366) 2024-09-21T01:04:06.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 -- 172.21.15.39:0/3761579801 learned_addr learned my addr 172.21.15.39:0/3761579801 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:06.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89400640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a3c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 -- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 msgr2=0x55c103a3c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a3c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 -- 172.21.15.39:0/3761579801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 msgr2=0x55c103a411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 -- 172.21.15.39:0/3761579801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c103a41730 con 0x55c1036480a0 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea8a800640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea89e00640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a3c1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0ad0b0 tx=0x7fea7c005d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6ece9037bdbf2a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea6be00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:04:06.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.078+0000 7fea6be00640 1 -- 172.21.15.39:0/3761579801 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7c020040 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea780044a0 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea6be00640 1 -- 172.21.15.39:0/3761579801 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea7c05c590 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea6be00640 1 -- 172.21.15.39:0/3761579801 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea7c0b1b50 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea97a7b740 10 client.25475 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c103a42a10 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea6be00640 1 -- 172.21.15.39:0/3761579801 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea780044a0 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea6be00640 1 -- 172.21.15.39:0/3761579801 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fea7c098170 con 0x55c1036480a0 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.079+0000 7fea97a7b740 10 client.25475 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.080+0000 7fea97a7b740 10 client.25475 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.080+0000 7fea97a7b740 20 client.25475 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.080+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c103604c90 0x55c103aa8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.080+0000 7fea97a7b740 4 client.25475 mds_command: new command op to 24742 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.080+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 --> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] -- command(tid 0: {"prefix": "status"}) -- 0x55c10398c750 con 0x55c103604c90 2024-09-21T01:04:06.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.080+0000 7fea89400640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c103604c90 0x55c103aa8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea89400640 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c103604c90 0x55c103aa8490 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8356556fa4af12fe server_cookie=6ae5e5aa8afc58ae in_seq=0 out_seq=0 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea6be00640 10 client.25475 ms_handle_connect on v2:172.21.15.160:6838/2403371273 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea6be00640 1 -- 172.21.15.39:0/3761579801 <== mds.2 v2:172.21.15.160:6838/2403371273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c10398c750 con 0x55c103604c90 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea6be00640 10 client.25475 handle_command_reply: tid=0 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 client.25475 shutdown 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 msgr2=0x55c103a3c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a3c1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0ad0b0 tx=0x7fea7c005d20 comp rx=0 tx=0).stop 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 shutdown_connections 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c103670f60 0x55c103a411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:6838/2403371273,v1:172.21.15.160:6839/2403371273] conn(0x55c103604c90 0x55c103aa8490 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c103650280 0x55c103a3c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 --2- 172.21.15.39:0/3761579801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c1036480a0 0x55c103a3c1a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea6be00640 0 client.25475 ms_handle_reset on v2:172.21.15.160:6838/2403371273 2024-09-21T01:04:06.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 >> 172.21.15.39:0/3761579801 conn(0x55c103665350 msgr2=0x55c103a275b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 shutdown_connections 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.081+0000 7fea97a7b740 1 -- 172.21.15.39:0/3761579801 wait complete. 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea97a7b740 20 client.25475 trim_cache size 0 max 16384 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 2, 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24742, 2024-09-21T01:04:06.086 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 147.74159736199999, 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 90, 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 226.73686762400001 2024-09-21T01:04:06.087 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea700b6700 msgr2=0x7fea700b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea700b6700 0x7fea700b8c10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fea84005fd0 tx=0x7fea84004960 comp rx=0 tx=0).stop 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c1047d0 msgr2=0x7fea8c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c1047d0 0x7fea8c19a270 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fea7c005fd0 tx=0x7fea7c00dde0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 shutdown_connections 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fea700b6700 0x7fea700b8c10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea8c10dc30 0x7fea8c19fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea8c105120 0x7fea8c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 --2- 172.21.15.39:0/3443335279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fea8c1047d0 0x7fea8c19a270 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.082+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 >> 172.21.15.39:0/3443335279 conn(0x7fea8c0fe2d0 msgr2=0x7fea8c100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.083+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 shutdown_connections 2024-09-21T01:04:06.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:06.083+0000 7fea8b200640 1 -- 172.21.15.39:0/3443335279 wait complete. 2024-09-21T01:04:06.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.a status 2024-09-21T01:04:07.303 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:07.637 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:07 smithi039 ceph-mon[30415]: pgmap v407: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-09-21T01:04:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:07 smithi150 ceph-mon[34945]: pgmap v407: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-09-21T01:04:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:07 smithi160 ceph-mon[34696]: pgmap v407: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 -- 172.21.15.39:0/83385914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 msgr2=0x7f7c3810fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 --2- 172.21.15.39:0/83385914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3810fa10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009900 tx=0x7f7c2402a410 comp rx=0 tx=0).stop 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 -- 172.21.15.39:0/83385914 shutdown_connections 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 --2- 172.21.15.39:0/83385914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c3810ff50 0x7f7c38112370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 --2- 172.21.15.39:0/83385914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3810fa10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 --2- 172.21.15.39:0/83385914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c38075750 0x7f7c38075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.963+0000 7f7c37e00640 1 -- 172.21.15.39:0/83385914 >> 172.21.15.39:0/83385914 conn(0x7f7c380ff3e0 msgr2=0x7f7c38101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.964+0000 7f7c37e00640 1 -- 172.21.15.39:0/83385914 shutdown_connections 2024-09-21T01:04:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.964+0000 7f7c37e00640 1 -- 172.21.15.39:0/83385914 wait complete. 2024-09-21T01:04:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.965+0000 7f7c37e00640 1 Processor -- start 2024-09-21T01:04:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.965+0000 7f7c37e00640 1 -- start start 2024-09-21T01:04:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c38075750 0x7f7c3819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c3810ff50 0x7f7c381a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7c38115ac0 con 0x7f7c380760a0 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c38115940 con 0x7f7c3810ff50 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7c38115c40 con 0x7f7c38075750 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c36000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c36a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c38075750 0x7f7c3819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c36000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49118/0 (socket says 172.21.15.39:49118) 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c36000640 1 -- 172.21.15.39:0/4087277045 learned_addr learned my addr 172.21.15.39:0/4087277045 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.966+0000 7f7c37400640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c3810ff50 0x7f7c381a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36a00640 1 -- 172.21.15.39:0/4087277045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c3810ff50 msgr2=0x7f7c381a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36a00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c3810ff50 0x7f7c381a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36a00640 1 -- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 msgr2=0x7f7c3819cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36a00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3819cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36a00640 1 -- 172.21.15.39:0/4087277045 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c381a2480 con 0x7f7c38075750 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36000640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3819cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:07.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c36a00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c38075750 0x7f7c3819c510 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800e9f0 tx=0x7f7c2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:07.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c3ca00640 1 -- 172.21.15.39:0/4087277045 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2800ccc0 con 0x7f7c38075750 2024-09-21T01:04:07.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c381a2770 con 0x7f7c38075750 2024-09-21T01:04:07.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c3ca00640 1 -- 172.21.15.39:0/4087277045 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c280108e0 con 0x7f7c38075750 2024-09-21T01:04:07.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.967+0000 7f7c3ca00640 1 -- 172.21.15.39:0/4087277045 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28018880 con 0x7f7c38075750 2024-09-21T01:04:07.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.968+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c381a8ed0 con 0x7f7c38075750 2024-09-21T01:04:07.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.970+0000 7f7c44a88740 1 Processor -- start 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 -- start start 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 0x5644787ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783dc0a0 0x5644787abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644783e4280 0x5644787ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564478722d50 con 0x5644783e4280 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x564478712180 con 0x564478404f60 2024-09-21T01:04:07.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c44a88740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x564478715f90 con 0x5644783dc0a0 2024-09-21T01:04:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c36000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644783e4280 0x5644787ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c37400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 0x5644787ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c37400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 0x5644787ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39376/0 (socket says 172.21.15.39:39376) 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c37400640 1 -- 172.21.15.39:0/4070071152 learned_addr learned my addr 172.21.15.39:0/4070071152 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.971+0000 7f7c36000640 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644783e4280 0x5644787ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49132/0 (socket says 172.21.15.39:49132) 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c3ca00640 1 -- 172.21.15.39:0/4087277045 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7c28010430 con 0x7f7c38075750 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c37400640 1 -- 172.21.15.39:0/4070071152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783dc0a0 msgr2=0x5644787abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c37400640 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783dc0a0 0x5644787abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c37400640 1 -- 172.21.15.39:0/4070071152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644783e4280 msgr2=0x5644787ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c37400640 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644783e4280 0x5644787ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c37400640 1 -- 172.21.15.39:0/4070071152 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644786dca70 con 0x564478404f60 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.972+0000 7f7c37400640 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 0x5644787ab670 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009fd0 tx=0x7f7c300118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=564f693415cb6429 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.973+0000 7f7c23e00640 1 -- 172.21.15.39:0/4070071152 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30004d20 con 0x564478404f60 2024-09-21T01:04:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.973+0000 7f7c23e00640 1 -- 172.21.15.39:0/4070071152 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c30004050 con 0x564478404f60 2024-09-21T01:04:07.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.973+0000 7f7c23e00640 1 -- 172.21.15.39:0/4070071152 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30005280 con 0x564478404f60 2024-09-21T01:04:07.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.973+0000 7f7c44a88740 1 -- 172.21.15.39:0/4070071152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 msgr2=0x5644787ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.973+0000 7f7c44a88740 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 0x5644787ab670 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009fd0 tx=0x7f7c300118d0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 -- 172.21.15.39:0/4070071152 shutdown_connections 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644783e4280 0x5644787ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783dc0a0 0x5644787abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 --2- 172.21.15.39:0/4070071152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564478404f60 0x5644787ab670 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 -- 172.21.15.39:0/4070071152 >> 172.21.15.39:0/4070071152 conn(0x5644783f9350 msgr2=0x5644787a9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c3ca00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c1c0b6680 0x7f7c1c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 -- 172.21.15.39:0/4070071152 shutdown_connections 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 -- 172.21.15.39:0/4070071152 wait complete. 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c3ca00640 1 -- 172.21.15.39:0/4087277045 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7c280e7e30 con 0x7f7c38075750 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c36000640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c1c0b6680 0x7f7c1c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.974+0000 7f7c44a88740 1 -- start start 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 Processor -- start 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c36000640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c1c0b6680 0x7f7c1c0b8b90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c381124a0 tx=0x7f7c24032ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787b7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5644785735f0 con 0x564478404f60 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5644786ad460 con 0x5644783dc0a0 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.975+0000 7f7c44a88740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5644787b9e50 con 0x5644783e4280 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c44a88740 10 client.0 _ll_register_callbacks cb 0x5644786bd210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52490/0 (socket says 172.21.15.39:52490) 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 -- 172.21.15.39:0/303329419 learned_addr learned my addr 172.21.15.39:0/303329419 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36a00640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 -- 172.21.15.39:0/303329419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 msgr2=0x5644787b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 -- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 msgr2=0x5644787c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787c3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36000640 1 -- 172.21.15.39:0/303329419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564478470540 con 0x5644783e4280 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c36a00640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.976+0000 7f7c37400640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c36000640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b77e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c2402a930 tx=0x7f7c24009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa9661e363a36fb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c20c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c20c00640 1 -- 172.21.15.39:0/303329419 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2404d070 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5644786afa70 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c20c00640 1 -- 172.21.15.39:0/303329419 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7c24005020 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c20c00640 1 -- 172.21.15.39:0/303329419 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c2405a050 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c44a88740 10 client.25477 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.977+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644786ca6f0 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c20c00640 1 -- 172.21.15.39:0/303329419 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5644786afa70 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c20c00640 1 -- 172.21.15.39:0/303329419 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7c240047a0 con 0x5644783e4280 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c44a88740 10 client.25477 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c44a88740 10 client.25477 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c44a88740 20 client.25477 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c44a88740 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5644783c38e0 0x5644787c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c44a88740 4 client.25477 mds_command: new command op to 24760 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 --> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564478724ca0 con 0x5644783c38e0 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.978+0000 7f7c36a00640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5644783c38e0 0x5644787c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.979+0000 7f7c36a00640 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5644783c38e0 0x5644787c9720 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd88b2be3cfefcc4 server_cookie=a15a8c657a06f6e4 in_seq=0 out_seq=0 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.979+0000 7f7c20c00640 10 client.25477 ms_handle_connect on v2:172.21.15.39:6840/3868264331 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c20c00640 1 -- 172.21.15.39:0/303329419 <== mds.1 v2:172.21.15.39:6840/3868264331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564478724ca0 con 0x5644783c38e0 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c20c00640 10 client.25477 handle_command_reply: tid=0 2024-09-21T01:04:07.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 client.25477 shutdown 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 msgr2=0x5644787b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b77e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c2402a930 tx=0x7f7c24009260 comp rx=0 tx=0).stop 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 shutdown_connections 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x5644783c38e0 0x5644787c9720 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b77e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 --2- 172.21.15.39:0/303329419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 >> 172.21.15.39:0/303329419 conn(0x5644783f9350 msgr2=0x5644787ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:07.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c20c00640 0 client.25477 ms_handle_reset on v2:172.21.15.39:6840/3868264331 2024-09-21T01:04:07.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 shutdown_connections 2024-09-21T01:04:07.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.980+0000 7f7c44a88740 1 -- 172.21.15.39:0/303329419 wait complete. 2024-09-21T01:04:07.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:07.981+0000 7f7c44a88740 20 client.25477 trim_cache size 0 max 16384 2024-09-21T01:04:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 Processor -- start 2024-09-21T01:04:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 -- start start 2024-09-21T01:04:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5644787c7a60 con 0x564478404f60 2024-09-21T01:04:08.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5644787c8130 con 0x5644783dc0a0 2024-09-21T01:04:08.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c44a88740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5644787c8800 con 0x5644783e4280 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49152/0 (socket says 172.21.15.39:49152) 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c37400640 1 -- 172.21.15.39:0/1854387757 learned_addr learned my addr 172.21.15.39:0/1854387757 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.000+0000 7f7c36a00640 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c36000640 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c37400640 1 -- 172.21.15.39:0/1854387757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 msgr2=0x5644787b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c37400640 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c37400640 1 -- 172.21.15.39:0/1854387757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 msgr2=0x5644787c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c37400640 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c37400640 1 -- 172.21.15.39:0/1854387757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644787ae4d0 con 0x564478404f60 2024-09-21T01:04:08.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.001+0000 7f7c37400640 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787b0e30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009250 tx=0x7f7c30009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fcc86b69a4b1ac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:08.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.002+0000 7f7c23400640 1 -- 172.21.15.39:0/1854387757 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30028070 con 0x564478404f60 2024-09-21T01:04:08.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.002+0000 7f7c23400640 1 -- 172.21.15.39:0/1854387757 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c30004d00 con 0x564478404f60 2024-09-21T01:04:08.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.002+0000 7f7c23400640 1 -- 172.21.15.39:0/1854387757 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30005440 con 0x564478404f60 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.002+0000 7f7c44a88740 1 -- 172.21.15.39:0/1854387757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 msgr2=0x5644787b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.002+0000 7f7c44a88740 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787b0e30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009250 tx=0x7f7c30009be0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 -- 172.21.15.39:0/1854387757 shutdown_connections 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787b0e30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 --2- 172.21.15.39:0/1854387757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 -- 172.21.15.39:0/1854387757 >> 172.21.15.39:0/1854387757 conn(0x5644783f9350 msgr2=0x5644787b7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 -- 172.21.15.39:0/1854387757 shutdown_connections 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 -- 172.21.15.39:0/1854387757 wait complete. 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.003+0000 7f7c44a88740 1 -- start start 2024-09-21T01:04:08.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 Processor -- start 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787ae970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5644787c95a0 con 0x564478404f60 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5644787c8ed0 con 0x5644783dc0a0 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5644787c9c70 con 0x5644783e4280 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c44a88740 10 client.0 _ll_register_callbacks cb 0x7f7c240317b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.004+0000 7f7c37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787aeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49162/0 (socket says 172.21.15.39:49162) 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c36a00640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c36000640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 -- 172.21.15.39:0/720897413 learned_addr learned my addr 172.21.15.39:0/720897413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 -- 172.21.15.39:0/720897413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 msgr2=0x5644787ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 -- 172.21.15.39:0/720897413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 msgr2=0x5644787d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 -- 172.21.15.39:0/720897413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644787af4a0 con 0x564478404f60 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c36000640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c37400640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787aeeb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7c30004620 tx=0x7f7c30004650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819e8a7b18b3490e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.005+0000 7f7c20c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c20c00640 1 -- 172.21.15.39:0/720897413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3003a060 con 0x564478404f60 2024-09-21T01:04:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c20c00640 1 -- 172.21.15.39:0/720897413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7c30004eb0 con 0x564478404f60 2024-09-21T01:04:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c30004d00 con 0x564478404f60 2024-09-21T01:04:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c20c00640 1 -- 172.21.15.39:0/720897413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c30028040 con 0x564478404f60 2024-09-21T01:04:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c44a88740 10 client.15474 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564478834bf0 con 0x564478404f60 2024-09-21T01:04:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.006+0000 7f7c20c00640 1 -- 172.21.15.39:0/720897413 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c30004d00 con 0x564478404f60 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c20c00640 1 -- 172.21.15.39:0/720897413 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7c3000f600 con 0x564478404f60 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c44a88740 10 client.15474 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c44a88740 10 client.15474 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c44a88740 20 client.15474 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c44a88740 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x564478398c90 0x564478837e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c44a88740 4 client.15474 mds_command: new command op to 24760 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 --> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] -- command(tid 0: {"prefix": "status"}) -- 0x564478720750 con 0x564478398c90 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.007+0000 7f7c36a00640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x564478398c90 0x564478837e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c36a00640 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x564478398c90 0x564478837e00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af351944d14a5ddb server_cookie=d51203a1cf145de0 in_seq=0 out_seq=0 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c20c00640 10 client.15474 ms_handle_connect on v2:172.21.15.39:6840/3868264331 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c20c00640 1 -- 172.21.15.39:0/720897413 <== mds.1 v2:172.21.15.39:6840/3868264331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564478720750 con 0x564478398c90 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c20c00640 10 client.15474 handle_command_reply: tid=0 2024-09-21T01:04:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 client.15474 shutdown 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 msgr2=0x5644787aeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787aeeb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7c30004620 tx=0x7f7c30004650 comp rx=0 tx=0).stop 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 shutdown_connections 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:6840/3868264331,v1:172.21.15.39:6841/3868264331] conn(0x564478398c90 0x564478837e00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564478404f60 0x5644787aeeb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5644783e4280 0x5644787ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 --2- 172.21.15.39:0/720897413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644783dc0a0 0x5644787d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 >> 172.21.15.39:0/720897413 conn(0x5644783f9350 msgr2=0x5644787b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.008+0000 7f7c20c00640 0 client.15474 ms_handle_reset on v2:172.21.15.39:6840/3868264331 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 shutdown_connections 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c44a88740 1 -- 172.21.15.39:0/720897413 wait complete. 2024-09-21T01:04:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c44a88740 20 client.15474 trim_cache size 0 max 16384 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24760, 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 150.682854564, 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 90, 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 221.06492531200001 2024-09-21T01:04:08.012 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c1c0b6680 msgr2=0x7f7c1c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c37e00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c1c0b6680 0x7f7c1c0b8b90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c381124a0 tx=0x7f7c24032ed0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c38075750 msgr2=0x7f7c3819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.009+0000 7f7c37e00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c38075750 0x7f7c3819c510 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800e9f0 tx=0x7f7c2800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 shutdown_connections 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7c1c0b6680 0x7f7c1c0b8b90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c3810ff50 0x7f7c381a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c380760a0 0x7f7c3819cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 --2- 172.21.15.39:0/4087277045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7c38075750 0x7f7c3819c510 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:08.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 >> 172.21.15.39:0/4087277045 conn(0x7f7c380ff3e0 msgr2=0x7f7c381089e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:08.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 shutdown_connections 2024-09-21T01:04:08.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:08.010+0000 7f7c37e00640 1 -- 172.21.15.39:0/4087277045 wait complete. 2024-09-21T01:04:09.044 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T01:04:09.423 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:09.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:09 smithi039 ceph-mon[30415]: pgmap v408: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:09 smithi150 ceph-mon[34945]: pgmap v408: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:09.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:09 smithi160 ceph-mon[34696]: pgmap v408: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 -- 172.21.15.39:0/2712085247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 msgr2=0x7f5cd0075b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 --2- 172.21.15.39:0/2712085247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd0075b60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009900 tx=0x7f5cc402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 -- 172.21.15.39:0/2712085247 shutdown_connections 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 --2- 172.21.15.39:0/2712085247 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5cd010ff50 0x7f5cd0112370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 --2- 172.21.15.39:0/2712085247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd00760a0 0x7f5cd010fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 --2- 172.21.15.39:0/2712085247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd0075b60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 -- 172.21.15.39:0/2712085247 >> 172.21.15.39:0/2712085247 conn(0x7f5cd00ff3e0 msgr2=0x7f5cd0101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:10.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 -- 172.21.15.39:0/2712085247 shutdown_connections 2024-09-21T01:04:10.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.047+0000 7f5cd6400640 1 -- 172.21.15.39:0/2712085247 wait complete. 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.048+0000 7f5cd6400640 1 Processor -- start 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.048+0000 7f5cd6400640 1 -- start start 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd6400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5cd00760a0 0x7f5cd019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd6400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd010ff50 0x7f5cd01a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd0115ac0 con 0x7f5cd0075750 2024-09-21T01:04:10.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd6400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd0115940 con 0x7f5cd010ff50 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd6400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd0115c40 con 0x7f5cd00760a0 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd019c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49172/0 (socket says 172.21.15.39:49172) 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd5a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd010ff50 0x7f5cd01a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5cd5000640 1 -- 172.21.15.39:0/3446617052 learned_addr learned my addr 172.21.15.39:0/3446617052 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.049+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5cd00760a0 0x7f5cd019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd5000640 1 -- 172.21.15.39:0/3446617052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5cd00760a0 msgr2=0x7f5cd019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5cd00760a0 0x7f5cd019ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd5000640 1 -- 172.21.15.39:0/3446617052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd010ff50 msgr2=0x7f5cd01a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd010ff50 0x7f5cd01a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd5000640 1 -- 172.21.15.39:0/3446617052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd01a2440 con 0x7f5cd0075750 2024-09-21T01:04:10.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd019c4d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005940 tx=0x7f5cc40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:10.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5ccd600640 1 -- 172.21.15.39:0/3446617052 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc403f030 con 0x7f5cd0075750 2024-09-21T01:04:10.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5ccd600640 1 -- 172.21.15.39:0/3446617052 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cc4003cb0 con 0x7f5cd0075750 2024-09-21T01:04:10.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd01a26d0 con 0x7f5cd0075750 2024-09-21T01:04:10.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5ccd600640 1 -- 172.21.15.39:0/3446617052 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4035070 con 0x7f5cd0075750 2024-09-21T01:04:10.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.050+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd01a8e90 con 0x7f5cd0075750 2024-09-21T01:04:10.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 Processor -- start 2024-09-21T01:04:10.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 -- start start 2024-09-21T01:04:10.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5ccd600640 1 -- 172.21.15.39:0/3446617052 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5cc4042030 con 0x7f5cd0075750 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121d8f60 0x563a1257f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b00a0 0x563a1257fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a125800f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563a124f6d50 con 0x563a121b00a0 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563a124e6180 con 0x563a121b8280 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.054+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563a124e9f90 con 0x563a121d8f60 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b00a0 0x563a1257fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a125800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5cd5a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121d8f60 0x563a1257f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a125800f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39422/0 (socket says 172.21.15.39:39422) 2024-09-21T01:04:10.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b00a0 0x563a1257fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49186/0 (socket says 172.21.15.39:49186) 2024-09-21T01:04:10.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3333555674 learned_addr learned my addr 172.21.15.39:0/3333555674 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3333555674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121d8f60 msgr2=0x563a1257f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121d8f60 0x563a1257f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3333555674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b00a0 msgr2=0x563a1257fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b00a0 0x563a1257fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.055+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3333555674 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a124b0a70 con 0x563a121b8280 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.056+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a125800f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00bc30 tx=0x7f5cbc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cd609d97ce9d6a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.056+0000 7f5cc2000640 1 -- 172.21.15.39:0/3333555674 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc01c480 con 0x563a121b8280 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.056+0000 7f5cc2000640 1 -- 172.21.15.39:0/3333555674 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cbc01cad0 con 0x563a121b8280 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.056+0000 7f5cc2000640 1 -- 172.21.15.39:0/3333555674 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc011af0 con 0x563a121b8280 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.056+0000 7f5ccd600640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5cb80b6700 0x7f5cb80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.056+0000 7f5ccd600640 1 -- 172.21.15.39:0/3446617052 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5cc4107f20 con 0x7f5cd0075750 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cd5a00640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5cb80b6700 0x7f5cb80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3333555674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 msgr2=0x563a125800f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a125800f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00bc30 tx=0x7f5cbc0107a0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3333555674 shutdown_connections 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a125800f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b00a0 0x563a1257fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3333555674 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121d8f60 0x563a1257f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3333555674 >> 172.21.15.39:0/3333555674 conn(0x563a121cd350 msgr2=0x563a1257d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3333555674 shutdown_connections 2024-09-21T01:04:10.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3333555674 wait complete. 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.057+0000 7f5cd5a00640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5cb80b6700 0x7f5cb80b8c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8005fd0 tx=0x7f5cc8009600 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.058+0000 7f5cde6c3740 1 -- start start 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.058+0000 7f5cde6c3740 1 Processor -- start 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a1258b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1258b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a12597170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563a123475f0 con 0x563a121d8f60 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563a12481460 con 0x563a121b8280 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563a1258de50 con 0x563a121b00a0 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cde6c3740 10 client.0 _ll_register_callbacks cb 0x563a12491210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a1258b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a12597170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a1258b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52560/0 (socket says 172.21.15.39:52560) 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 -- 172.21.15.39:0/2429034452 learned_addr learned my addr 172.21.15.39:0/2429034452 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:10.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1258b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 -- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 msgr2=0x563a1258b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1258b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 -- 172.21.15.39:0/2429034452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 msgr2=0x563a12597170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a12597170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.059+0000 7f5cd5000640 1 -- 172.21.15.39:0/2429034452 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a12244540 con 0x563a121b00a0 2024-09-21T01:04:10.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cd5000640 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a1258b1d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4006010 tx=0x7f5cc402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5e354ba6815bbd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cb6a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cb6a00640 1 -- 172.21.15.39:0/2429034452 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4049040 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a12483a70 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cb6a00640 1 -- 172.21.15.39:0/2429034452 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5cc40d6f20 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cb6a00640 1 -- 172.21.15.39:0/2429034452 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cc40c98e0 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cde6c3740 10 client.25495 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.060+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a1249e6f0 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cb6a00640 1 -- 172.21.15.39:0/2429034452 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563a12483a70 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cb6a00640 1 -- 172.21.15.39:0/2429034452 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5cc410b020 con 0x563a121b00a0 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cde6c3740 10 client.25495 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cde6c3740 10 client.25495 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cde6c3740 20 client.25495 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:10.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a121978e0 0x563a1259d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cde6c3740 4 client.25495 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:04:10.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.061+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a124f8ca0 con 0x563a121978e0 2024-09-21T01:04:10.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.062+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a121978e0 0x563a1259d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.062+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a121978e0 0x563a1259d6d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=368bfc8566b00a57 server_cookie=90d80075e6aee43a in_seq=0 out_seq=0 2024-09-21T01:04:10.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.062+0000 7f5cb6a00640 10 client.25495 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cb6a00640 1 -- 172.21.15.39:0/2429034452 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563a124f8ca0 con 0x563a121978e0 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cb6a00640 10 client.25495 handle_command_reply: tid=0 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 client.25495 shutdown 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 msgr2=0x563a1258b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a1258b1d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4006010 tx=0x7f5cc402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 shutdown_connections 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a121978e0 0x563a1259d6d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a12597170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1258b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/2429034452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a1258b1d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 >> 172.21.15.39:0/2429034452 conn(0x563a121cd350 msgr2=0x563a12582800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:10.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.063+0000 7f5cb6a00640 0 client.25495 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:10.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.064+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 shutdown_connections 2024-09-21T01:04:10.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.064+0000 7f5cde6c3740 1 -- 172.21.15.39:0/2429034452 wait complete. 2024-09-21T01:04:10.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.064+0000 7f5cde6c3740 20 client.25495 trim_cache size 0 max 16384 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 Processor -- start 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 -- start start 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125970e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 0x563a12597620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121d8f60 0x563a12586180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563a1259ba10 con 0x563a121b8280 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563a1259c0e0 con 0x563a121d8f60 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.083+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563a1259c7b0 con 0x563a121b00a0 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 0x563a12597620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 0x563a12597620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49202/0 (socket says 172.21.15.39:49202) 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3954916914 learned_addr learned my addr 172.21.15.39:0/3954916914 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5cd5a00640 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121d8f60 0x563a12586180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3954916914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 msgr2=0x563a125970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3954916914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121d8f60 msgr2=0x563a12586180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121d8f60 0x563a12586180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 -- 172.21.15.39:0/3954916914 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1258bb70 con 0x563a121b8280 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.084+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 0x563a12597620 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005890 tx=0x7f5cbc01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff805d150ac0c0ef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.085+0000 7f5cc1600640 1 -- 172.21.15.39:0/3954916914 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc005240 con 0x563a121b8280 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.085+0000 7f5cc1600640 1 -- 172.21.15.39:0/3954916914 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cbc0053e0 con 0x563a121b8280 2024-09-21T01:04:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.085+0000 7f5cc1600640 1 -- 172.21.15.39:0/3954916914 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc01c480 con 0x563a121b8280 2024-09-21T01:04:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3954916914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 msgr2=0x563a12597620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 0x563a12597620 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005890 tx=0x7f5cbc01e040 comp rx=0 tx=0).stop 2024-09-21T01:04:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3954916914 shutdown_connections 2024-09-21T01:04:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121d8f60 0x563a12586180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121b8280 0x563a12597620 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3954916914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3954916914 >> 172.21.15.39:0/3954916914 conn(0x563a121cd350 msgr2=0x563a1259af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3954916914 shutdown_connections 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3954916914 wait complete. 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.086+0000 7f5cde6c3740 1 -- start start 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 Processor -- start 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1259f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a1259f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563a1259d550 con 0x563a121d8f60 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563a1259ce80 con 0x563a121b8280 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563a1259dc20 con 0x563a121b00a0 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cde6c3740 10 client.0 _ll_register_callbacks cb 0x7f5cc4003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cd5000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52580/0 (socket says 172.21.15.39:52580) 2024-09-21T01:04:10.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cd5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a1259f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.087+0000 7f5cd5000640 1 -- 172.21.15.39:0/3797436713 learned_addr learned my addr 172.21.15.39:0/3797436713 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:10.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1259f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cd5000640 1 -- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 msgr2=0x563a1259f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1259f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cd5000640 1 -- 172.21.15.39:0/3797436713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 msgr2=0x563a1259f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a1259f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cd5000640 1 -- 172.21.15.39:0/3797436713 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1259ff00 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cd5000640 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125a5f50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5cc410bb60 tx=0x7f5cc410b950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b056535525b6866 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cb6a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cb6a00640 1 -- 172.21.15.39:0/3797436713 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4049040 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cb6a00640 1 -- 172.21.15.39:0/3797436713 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5cc4003800 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.088+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cbc0053e0 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cb6a00640 1 -- 172.21.15.39:0/3797436713 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cc40c91b0 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 10 client.25501 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a1259ff00 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cb6a00640 1 -- 172.21.15.39:0/3797436713 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cbc0053e0 con 0x563a121b00a0 2024-09-21T01:04:10.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cb6a00640 1 -- 172.21.15.39:0/3797436713 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5cc40ccca0 con 0x563a121b00a0 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 10 client.25501 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 10 client.25501 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 20 client.25501 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a1216cc90 0x563a1260bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 4 client.25501 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.089+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "status"}) -- 0x563a124f4750 con 0x563a1216cc90 2024-09-21T01:04:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.090+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a1216cc90 0x563a1260bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.090+0000 7f5ccfe00640 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a1216cc90 0x563a1260bde0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6257e2ff66951d5 server_cookie=bbb661bc16be83ba in_seq=0 out_seq=0 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.090+0000 7f5cb6a00640 10 client.25501 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cb6a00640 1 -- 172.21.15.39:0/3797436713 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563a124f4750 con 0x563a1216cc90 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cb6a00640 10 client.25501 handle_command_reply: tid=0 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 client.25501 shutdown 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 msgr2=0x563a125a5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125a5f50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5cc410bb60 tx=0x7f5cc410b950 comp rx=0 tx=0).stop 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 shutdown_connections 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563a1216cc90 0x563a1260bde0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a121d8f60 0x563a1259f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563a121b8280 0x563a1259f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 --2- 172.21.15.39:0/3797436713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563a121b00a0 0x563a125a5f50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cb6a00640 0 client.25501 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 >> 172.21.15.39:0/3797436713 conn(0x563a121cd350 msgr2=0x563a1259af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 shutdown_connections 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 1 -- 172.21.15.39:0/3797436713 wait complete. 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.091+0000 7f5cde6c3740 20 client.25501 trim_cache size 0 max 16384 2024-09-21T01:04:10.094 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "61abfb08-77b3-11ef-baf5-efdc52797490", 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24764, 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 205.59072638699999, 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 90, 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 90, 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 220.604986293 2024-09-21T01:04:10.095 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5cb80b6700 msgr2=0x7f5cb80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5cb80b6700 0x7f5cb80b8c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8005fd0 tx=0x7f5cc8009600 comp rx=0 tx=0).stop 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 msgr2=0x7f5cd019c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd019c4d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005940 tx=0x7f5cc40042f0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 shutdown_connections 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5cb80b6700 0x7f5cb80b8c10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd010ff50 0x7f5cd01a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5cd00760a0 0x7f5cd019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 --2- 172.21.15.39:0/3446617052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cd0075750 0x7f5cd019c4d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 >> 172.21.15.39:0/3446617052 conn(0x7f5cd00ff3e0 msgr2=0x7f5cd0109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 shutdown_connections 2024-09-21T01:04:10.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:10.092+0000 7f5cd6400640 1 -- 172.21.15.39:0/3446617052 wait complete. 2024-09-21T01:04:10.945 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-21T01:04:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:11 smithi039 ceph-mon[30415]: pgmap v409: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:11 smithi150 ceph-mon[34945]: pgmap v409: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:11 smithi160 ceph-mon[34696]: pgmap v409: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:13 smithi039 ceph-mon[30415]: pgmap v410: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:13 smithi150 ceph-mon[34945]: pgmap v410: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:13 smithi160 ceph-mon[34696]: pgmap v410: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-09-21T01:04:15.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:15 smithi039 ceph-mon[30415]: pgmap v411: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-09-21T01:04:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:15 smithi150 ceph-mon[34945]: pgmap v411: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-09-21T01:04:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:15 smithi160 ceph-mon[34696]: pgmap v411: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-09-21T01:04:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:17 smithi150 ceph-mon[34945]: pgmap v412: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T01:04:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:17 smithi160 ceph-mon[34696]: pgmap v412: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T01:04:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:17 smithi039 ceph-mon[30415]: pgmap v412: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T01:04:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:19 smithi150 ceph-mon[34945]: pgmap v413: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:04:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:19 smithi160 ceph-mon[34696]: pgmap v413: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:04:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:19 smithi039 ceph-mon[30415]: pgmap v413: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:04:21.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:21 smithi150 ceph-mon[34945]: pgmap v414: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:21 smithi160 ceph-mon[34696]: pgmap v414: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:21 smithi039 ceph-mon[30415]: pgmap v414: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:23 smithi150 ceph-mon[34945]: pgmap v415: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:04:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:23 smithi160 ceph-mon[34696]: pgmap v415: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:04:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:23 smithi039 ceph-mon[30415]: pgmap v415: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-21T01:04:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:25 smithi150 ceph-mon[34945]: pgmap v416: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:04:25.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:25 smithi160 ceph-mon[34696]: pgmap v416: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:04:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:25 smithi039 ceph-mon[30415]: pgmap v416: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:04:25.947 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.5 2024-09-21T01:04:25.947 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:04:26.356 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 -- 172.21.15.39:0/2202922677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c075730 msgr2=0x7f101c075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 --2- 172.21.15.39:0/2202922677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c075730 0x7f101c075b40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f100c009900 tx=0x7f100c02a410 comp rx=0 tx=0).stop 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 -- 172.21.15.39:0/2202922677 shutdown_connections 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 --2- 172.21.15.39:0/2202922677 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f101c110140 0x7f101c112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 --2- 172.21.15.39:0/2202922677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c076080 0x7f101c10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 --2- 172.21.15.39:0/2202922677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c075730 0x7f101c075b40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 -- 172.21.15.39:0/2202922677 >> 172.21.15.39:0/2202922677 conn(0x7f101c07b850 msgr2=0x7f101c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:27.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 -- 172.21.15.39:0/2202922677 shutdown_connections 2024-09-21T01:04:27.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.031+0000 7f1022e00640 1 -- 172.21.15.39:0/2202922677 wait complete. 2024-09-21T01:04:27.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.032+0000 7f1022e00640 1 Processor -- start 2024-09-21T01:04:27.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.032+0000 7f1022e00640 1 -- start start 2024-09-21T01:04:27.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1022e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 0x7f101c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1022e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c076080 0x7f101c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1022e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f101c110140 0x7f101c1a2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1022e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f101c115cd0 con 0x7f101c076080 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1022e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f101c115b50 con 0x7f101c075730 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1022e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f101c115e50 con 0x7f101c110140 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f101be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 0x7f101c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1013e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c076080 0x7f101c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f101be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 0x7f101c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:55816/0 (socket says 172.21.15.39:55816) 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f101be00640 1 -- 172.21.15.39:0/3369996469 learned_addr learned my addr 172.21.15.39:0/3369996469 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.033+0000 7f1020e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f101c110140 0x7f101c1a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.034+0000 7f101be00640 1 -- 172.21.15.39:0/3369996469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f101c110140 msgr2=0x7f101c1a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.034+0000 7f101be00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f101c110140 0x7f101c1a2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.034+0000 7f101be00640 1 -- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c076080 msgr2=0x7f101c19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.034+0000 7f101be00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c076080 0x7f101c19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.034+0000 7f101be00640 1 -- 172.21.15.39:0/3369996469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c1a2750 con 0x7f101c075730 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.034+0000 7f101be00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 0x7f101c19c7e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f100c005ab0 tx=0x7f100c003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.035+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c03f070 con 0x7f101c075730 2024-09-21T01:04:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.035+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c1a29e0 con 0x7f101c075730 2024-09-21T01:04:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.035+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c031d60 con 0x7f101c075730 2024-09-21T01:04:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.035+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c035030 con 0x7f101c075730 2024-09-21T01:04:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.035+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c1a91a0 con 0x7f101c075730 2024-09-21T01:04:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.036+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c075b40 con 0x7f101c075730 2024-09-21T01:04:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.039+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f100c042070 con 0x7f101c075730 2024-09-21T01:04:27.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.041+0000 7f1019600640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f10000b6680 0x7f10000b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:27.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.041+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f100c1088a0 con 0x7f101c075730 2024-09-21T01:04:27.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.041+0000 7f1013e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f10000b6680 0x7f10000b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:27.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.042+0000 7f1013e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f10000b6680 0x7f10000b8b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f101c111eb0 tx=0x7f100400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:27.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.042+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f100c0d2df0 con 0x7f101c075730 2024-09-21T01:04:27.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.311+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f101c19d910 con 0x7f101c075730 2024-09-21T01:04:27.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.312+0000 7f1019600640 1 -- 172.21.15.39:0/3369996469 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7f100c0d7d30 con 0x7f101c075730 2024-09-21T01:04:27.314 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:04:27.314 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:04:27.314 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:04:27.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f10000b6680 msgr2=0x7f10000b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:27.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f10000b6680 0x7f10000b8b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f101c111eb0 tx=0x7f100400b390 comp rx=0 tx=0).stop 2024-09-21T01:04:27.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 msgr2=0x7f101c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:27.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 0x7f101c19c7e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f100c005ab0 tx=0x7f100c003c80 comp rx=0 tx=0).stop 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 shutdown_connections 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f101c110140 0x7f101c1a2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f101c076080 0x7f101c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f10000b6680 0x7f10000b8b90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 --2- 172.21.15.39:0/3369996469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c075730 0x7f101c19c7e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 >> 172.21.15.39:0/3369996469 conn(0x7f101c07b850 msgr2=0x7f101c108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 shutdown_connections 2024-09-21T01:04:27.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:27.316+0000 7f1022e00640 1 -- 172.21.15.39:0/3369996469 wait complete. 2024-09-21T01:04:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:27 smithi039 ceph-mon[30415]: pgmap v417: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:27 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3369996469' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:27 smithi150 ceph-mon[34945]: pgmap v417: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:27.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:27 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3369996469' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:27 smithi160 ceph-mon[34696]: pgmap v417: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:27 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3369996469' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:28.181 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 134.5 --await-for 45.0' 2024-09-21T01:04:28.182 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 134.5 --await-for 45.0 2024-09-21T01:04:28.586 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.347+0000 7f0a50a00640 1 -- 172.21.15.39:0/3568156009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 msgr2=0x7f0a4c075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.347+0000 7f0a50a00640 1 --2- 172.21.15.39:0/3568156009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c075bc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009900 tx=0x7f0a3c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 -- 172.21.15.39:0/3568156009 shutdown_connections 2024-09-21T01:04:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 --2- 172.21.15.39:0/3568156009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a4c10ff20 0x7f0a4c112360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 --2- 172.21.15.39:0/3568156009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c10f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 --2- 172.21.15.39:0/3568156009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c075bc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 -- 172.21.15.39:0/3568156009 >> 172.21.15.39:0/3568156009 conn(0x7f0a4c0ff430 msgr2=0x7f0a4c101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 -- 172.21.15.39:0/3568156009 shutdown_connections 2024-09-21T01:04:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.348+0000 7f0a50a00640 1 -- 172.21.15.39:0/3568156009 wait complete. 2024-09-21T01:04:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.349+0000 7f0a50a00640 1 Processor -- start 2024-09-21T01:04:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.349+0000 7f0a50a00640 1 -- start start 2024-09-21T01:04:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a50a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a50a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a50a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a4c10ff20 0x7f0a4c1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a50a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c115b20 con 0x7f0a4c0757b0 2024-09-21T01:04:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a50a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1159a0 con 0x7f0a4c10ff20 2024-09-21T01:04:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a50a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c115ca0 con 0x7f0a4c076100 2024-09-21T01:04:29.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a4aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a4aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:56256/0 (socket says 172.21.15.39:56256) 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a4b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a4aa00640 1 -- 172.21.15.39:0/662334030 learned_addr learned my addr 172.21.15.39:0/662334030 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.350+0000 7f0a4be00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a4c10ff20 0x7f0a4c1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4aa00640 1 -- 172.21.15.39:0/662334030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a4c10ff20 msgr2=0x7f0a4c1a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a4c10ff20 0x7f0a4c1a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4aa00640 1 -- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 msgr2=0x7f0a4c19c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c19c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4aa00640 1 -- 172.21.15.39:0/662334030 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c1a2420 con 0x7f0a4c076100 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4b400640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c19ca50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800e9c0 tx=0x7f0a3800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.351+0000 7f0a43e00640 1 -- 172.21.15.39:0/662334030 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3800cd80 con 0x7f0a4c076100 2024-09-21T01:04:29.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.352+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c1a2710 con 0x7f0a4c076100 2024-09-21T01:04:29.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.352+0000 7f0a43e00640 1 -- 172.21.15.39:0/662334030 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a380035f0 con 0x7f0a4c076100 2024-09-21T01:04:29.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.352+0000 7f0a43e00640 1 -- 172.21.15.39:0/662334030 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a38010690 con 0x7f0a4c076100 2024-09-21T01:04:29.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.352+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c1a8e70 con 0x7f0a4c076100 2024-09-21T01:04:29.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.356+0000 7f0a58c8c740 1 Processor -- start 2024-09-21T01:04:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.356+0000 7f0a43e00640 1 -- 172.21.15.39:0/662334030 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f0a380037a0 con 0x7f0a4c076100 2024-09-21T01:04:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.356+0000 7f0a58c8c740 1 -- start start 2024-09-21T01:04:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56059903dca0 con 0x560598d20840 2024-09-21T01:04:29.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56059902d0d0 con 0x560598cffb60 2024-09-21T01:04:29.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560599030ee0 con 0x560598cf7980 2024-09-21T01:04:29.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.357+0000 7f0a4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50924/0 (socket says 172.21.15.39:50924) 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 -- 172.21.15.39:0/475073161 learned_addr learned my addr 172.21.15.39:0/475073161 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4b400640 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 -- 172.21.15.39:0/475073161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 msgr2=0x5605990c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 -- 172.21.15.39:0/475073161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 msgr2=0x5605990c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4be00640 1 -- 172.21.15.39:0/475073161 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560598fdf400 con 0x560598d20840 2024-09-21T01:04:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.358+0000 7f0a43e00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a300b6670 0x7f0a300b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.359+0000 7f0a4be00640 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990c8510 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009fd0 tx=0x7f0a440118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cda8878cb5319 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.359+0000 7f0a43e00640 1 -- 172.21.15.39:0/662334030 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0a380e7ea0 con 0x7f0a4c076100 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.359+0000 7f0a4b400640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a300b6670 0x7f0a300b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.359+0000 7f0a37e00640 1 -- 172.21.15.39:0/475073161 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44004d20 con 0x560598d20840 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.359+0000 7f0a37e00640 1 -- 172.21.15.39:0/475073161 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a44004050 con 0x560598d20840 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.359+0000 7f0a37e00640 1 -- 172.21.15.39:0/475073161 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44005280 con 0x560598d20840 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 -- 172.21.15.39:0/475073161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 msgr2=0x5605990c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a4b400640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a300b6670 0x7f0a300b8b80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000c00 tx=0x7f0a3c009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990c8510 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009fd0 tx=0x7f0a440118d0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 -- 172.21.15.39:0/475073161 shutdown_connections 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/475073161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990c8510 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.360+0000 7f0a58c8c740 1 -- 172.21.15.39:0/475073161 >> 172.21.15.39:0/475073161 conn(0x560598d14990 msgr2=0x5605990c6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.361+0000 7f0a58c8c740 1 -- 172.21.15.39:0/475073161 shutdown_connections 2024-09-21T01:04:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.361+0000 7f0a58c8c740 1 -- 172.21.15.39:0/475073161 wait complete. 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.361+0000 7f0a58c8c740 1 -- start start 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.362+0000 7f0a58c8c740 1 Processor -- start 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.362+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cf7980 0x5605990d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.362+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 0x5605990d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990d4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560598e8e7a0 con 0x560598d20840 2024-09-21T01:04:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560598fd2a10 con 0x560598cf7980 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5605990d6cf0 con 0x560598cffb60 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a58c8c740 10 client.0 _ll_register_callbacks cb 0x560598fe2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990d4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 0x5605990d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990d4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50936/0 (socket says 172.21.15.39:50936) 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4be00640 1 -- 172.21.15.39:0/3778994752 learned_addr learned my addr 172.21.15.39:0/3778994752 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 0x5605990d4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:56280/0 (socket says 172.21.15.39:56280) 2024-09-21T01:04:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4b400640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cf7980 0x5605990d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4aa00640 1 -- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cf7980 msgr2=0x5605990d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cf7980 0x5605990d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4aa00640 1 -- 172.21.15.39:0/3778994752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 msgr2=0x5605990d4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.363+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990d4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a4aa00640 1 -- 172.21.15.39:0/3778994752 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560598d9ab90 con 0x560598cffb60 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a4be00640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990d4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 0x5605990d4080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a380e5270 tx=0x7f0a380a9fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3f086a2186f9d2f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a34c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a34c00640 1 -- 172.21.15.39:0/3778994752 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a380ad2c0 con 0x560598cffb60 2024-09-21T01:04:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a44004050 con 0x560598cffb60 2024-09-21T01:04:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.364+0000 7f0a34c00640 1 -- 172.21.15.39:0/3778994752 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0a380a92c0 con 0x560598cffb60 2024-09-21T01:04:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a34c00640 1 -- 172.21.15.39:0/3778994752 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a380a90e0 con 0x560598cffb60 2024-09-21T01:04:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 10 client.25519 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5605990e29f0 con 0x560598cffb60 2024-09-21T01:04:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a34c00640 1 -- 172.21.15.39:0/3778994752 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a38098020 con 0x560598cffb60 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a34c00640 1 -- 172.21.15.39:0/3778994752 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0a380d2030 con 0x560598cffb60 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 10 client.25519 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 10 client.25519 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 20 client.25519 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cdf1c0 0x5605990e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 4 client.25519 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:04:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.365+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56059903fbf0 con 0x560598cdf1c0 2024-09-21T01:04:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.366+0000 7f0a4b400640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cdf1c0 0x5605990e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.367+0000 7f0a4b400640 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cdf1c0 0x5605990e5ec0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6880dd4fa6815aeb server_cookie=85b7d7b4ba880c61 in_seq=0 out_seq=0 2024-09-21T01:04:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.367+0000 7f0a34c00640 10 client.25519 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a34c00640 1 -- 172.21.15.39:0/3778994752 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56059903fbf0 con 0x560598cdf1c0 2024-09-21T01:04:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a34c00640 10 client.25519 handle_command_reply: tid=0 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 client.25519 shutdown 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 msgr2=0x5605990d4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 0x5605990d4080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a380e5270 tx=0x7f0a380a9fc0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 shutdown_connections 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cdf1c0 0x5605990e5ec0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x5605990d4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cffb60 0x5605990d4080 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a34c00640 0 client.25519 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/3778994752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cf7980 0x5605990d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 >> 172.21.15.39:0/3778994752 conn(0x560598d14990 msgr2=0x5605990cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.368+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 shutdown_connections 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.369+0000 7f0a58c8c740 1 -- 172.21.15.39:0/3778994752 wait complete. 2024-09-21T01:04:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.369+0000 7f0a58c8c740 20 client.25519 trim_cache size 0 max 16384 2024-09-21T01:04:29.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 Processor -- start 2024-09-21T01:04:29.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 -- start start 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 0x5605990c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598d20840 0x5605990d5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1b57d0 con 0x560598cf7980 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1b30a0 con 0x560598cffb60 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.388+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1b0ef0 con 0x560598d20840 2024-09-21T01:04:29.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 0x5605990c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 0x5605990c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50946/0 (socket says 172.21.15.39:50946) 2024-09-21T01:04:29.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 -- 172.21.15.39:0/2634139929 learned_addr learned my addr 172.21.15.39:0/2634139929 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:29.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4be00640 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598d20840 0x5605990d5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 -- 172.21.15.39:0/2634139929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598d20840 msgr2=0x5605990d5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598d20840 0x5605990d5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 -- 172.21.15.39:0/2634139929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 msgr2=0x5605990d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4b400640 1 -- 172.21.15.39:0/2634139929 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605990c71c0 con 0x560598cf7980 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.389+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.390+0000 7f0a4b400640 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 0x5605990c9bb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009900 tx=0x7f0a3c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff28522563b85e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.390+0000 7f0a37400640 1 -- 172.21.15.39:0/2634139929 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c035040 con 0x560598cf7980 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.390+0000 7f0a37400640 1 -- 172.21.15.39:0/2634139929 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a3c004d10 con 0x560598cf7980 2024-09-21T01:04:29.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.390+0000 7f0a37400640 1 -- 172.21.15.39:0/2634139929 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c032710 con 0x560598cf7980 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.391+0000 7f0a58c8c740 1 -- 172.21.15.39:0/2634139929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 msgr2=0x5605990c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.391+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 0x5605990c9bb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009900 tx=0x7f0a3c03e000 comp rx=0 tx=0).stop 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 -- 172.21.15.39:0/2634139929 shutdown_connections 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598d20840 0x5605990d5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/2634139929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598cf7980 0x5605990c9bb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 -- 172.21.15.39:0/2634139929 >> 172.21.15.39:0/2634139929 conn(0x560598d14990 msgr2=0x5605990deed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:29.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 -- 172.21.15.39:0/2634139929 shutdown_connections 2024-09-21T01:04:29.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 -- 172.21.15.39:0/2634139929 wait complete. 2024-09-21T01:04:29.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.392+0000 7f0a58c8c740 1 -- start start 2024-09-21T01:04:29.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 Processor -- start 2024-09-21T01:04:29.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990d8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x560599158070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1b1560 con 0x560598d20840 2024-09-21T01:04:29.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1b29d0 con 0x560598cffb60 2024-09-21T01:04:29.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1b1c30 con 0x560598cf7980 2024-09-21T01:04:29.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a58c8c740 10 client.0 _ll_register_callbacks cb 0x7f0a380046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.393+0000 7f0a4aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990d8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:56292/0 (socket says 172.21.15.39:56292) 2024-09-21T01:04:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:55872/0 (socket says 172.21.15.39:55872) 2024-09-21T01:04:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 -- 172.21.15.39:0/387346876 learned_addr learned my addr 172.21.15.39:0/387346876 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4be00640 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x560599158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 -- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 msgr2=0x5605990d8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 -- 172.21.15.39:0/387346876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 msgr2=0x560599158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x560599158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 -- 172.21.15.39:0/387346876 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605991585b0 con 0x560598cf7980 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a4b400640 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990d8680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c006010 tx=0x7f0a3c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a894697cb0b017 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.394+0000 7f0a34c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.395+0000 7f0a34c00640 1 -- 172.21.15.39:0/387346876 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c035040 con 0x560598cf7980 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.395+0000 7f0a34c00640 1 -- 172.21.15.39:0/387346876 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0a3c004b10 con 0x560598cf7980 2024-09-21T01:04:29.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.395+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a3c004d10 con 0x560598cf7980 2024-09-21T01:04:29.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.395+0000 7f0a34c00640 1 -- 172.21.15.39:0/387346876 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a3c003800 con 0x560598cf7980 2024-09-21T01:04:29.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.395+0000 7f0a58c8c740 10 client.25525 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:29.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.395+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5605991597e0 con 0x560598cf7980 2024-09-21T01:04:29.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a34c00640 1 -- 172.21.15.39:0/387346876 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a3c004d10 con 0x560598cf7980 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a34c00640 1 -- 172.21.15.39:0/387346876 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0a3c059020 con 0x560598cf7980 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a58c8c740 10 client.25525 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a58c8c740 10 client.25525 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a58c8c740 20 client.25525 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cb4570 0x56059915bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a58c8c740 4 client.25525 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.5, "await_for": 45.0}] 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.396+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.5, "await_for": 45.0}) -- 0x56059903b6a0 con 0x560598cb4570 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.397+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cb4570 0x56059915bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.397+0000 7f0a4aa00640 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cb4570 0x56059915bf90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc5f8d7c590869e7 server_cookie=f7573e232407f09c in_seq=0 out_seq=0 2024-09-21T01:04:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.397+0000 7f0a34c00640 10 client.25525 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a34c00640 1 -- 172.21.15.39:0/387346876 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56059903b6a0 con 0x560598cb4570 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a34c00640 10 client.25525 handle_command_reply: tid=0 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 client.25525 shutdown 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 msgr2=0x5605990d8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990d8680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c006010 tx=0x7f0a3c005740 comp rx=0 tx=0).stop 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 shutdown_connections 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560598cb4570 0x56059915bf90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560598d20840 0x560599158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560598cffb60 0x5605990d8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 --2- 172.21.15.39:0/387346876 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560598cf7980 0x5605990d8680 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a34c00640 0 client.25525 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.409+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 >> 172.21.15.39:0/387346876 conn(0x560598d14990 msgr2=0x5605990cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.410+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 shutdown_connections 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.410+0000 7f0a58c8c740 1 -- 172.21.15.39:0/387346876 wait complete. 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.410+0000 7f0a58c8c740 20 client.25525 trim_cache size 0 max 16384 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 27, 2024-09-21T01:04:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 2, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "fe47de4d": { 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 2, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 134.5, 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:04:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:29.414 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a300b6670 msgr2=0x7f0a300b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a300b6670 0x7f0a300b8b80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000c00 tx=0x7f0a3c009560 comp rx=0 tx=0).stop 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 msgr2=0x7f0a4c19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c19ca50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800e9c0 tx=0x7f0a3800ee80 comp rx=0 tx=0).stop 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 shutdown_connections 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0a300b6670 0x7f0a300b8b80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a4c10ff20 0x7f0a4c1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0a4c076100 0x7f0a4c19ca50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 --2- 172.21.15.39:0/662334030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4c0757b0 0x7f0a4c19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 >> 172.21.15.39:0/662334030 conn(0x7f0a4c0ff430 msgr2=0x7f0a4c100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 shutdown_connections 2024-09-21T01:04:29.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:29.411+0000 7f0a50a00640 1 -- 172.21.15.39:0/662334030 wait complete. 2024-09-21T01:04:29.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:29 smithi039 ceph-mon[30415]: pgmap v418: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:04:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:29 smithi150 ceph-mon[34945]: pgmap v418: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:04:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:29 smithi160 ceph-mon[34696]: pgmap v418: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:04:30.285 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fe47de4d', quiesce took 4.3 seconds. Will release after: 10.2 2024-09-21T01:04:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:31 smithi150 ceph-mon[34945]: pgmap v419: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:31 smithi160 ceph-mon[34696]: pgmap v419: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:31 smithi039 ceph-mon[30415]: pgmap v419: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:33 smithi150 ceph-mon[34945]: pgmap v420: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:33.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:33 smithi160 ceph-mon[34696]: pgmap v420: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:33 smithi039 ceph-mon[30415]: pgmap v420: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:35 smithi150 ceph-mon[34945]: pgmap v421: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-21T01:04:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:35 smithi160 ceph-mon[34696]: pgmap v421: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-21T01:04:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:35 smithi039 ceph-mon[30415]: pgmap v421: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-21T01:04:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:37 smithi150 ceph-mon[34945]: pgmap v422: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:37 smithi160 ceph-mon[34696]: pgmap v422: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:37 smithi039 ceph-mon[30415]: pgmap v422: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:39 smithi150 ceph-mon[34945]: pgmap v423: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:39 smithi160 ceph-mon[34696]: pgmap v423: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:39 smithi039 ceph-mon[30415]: pgmap v423: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:40.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:04:40.968 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.695+0000 7fc9cda00640 1 -- 172.21.15.39:0/3661454453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 msgr2=0x7fc9c8110110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.695+0000 7fc9cda00640 1 --2- 172.21.15.39:0/3661454453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c8110110 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009ca0 tx=0x7fc9c002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 -- 172.21.15.39:0/3661454453 shutdown_connections 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 --2- 172.21.15.39:0/3661454453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c8110110 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 --2- 172.21.15.39:0/3661454453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c8105110 0x7fc9c810d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 --2- 172.21.15.39:0/3661454453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9c81047c0 0x7fc9c8104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 -- 172.21.15.39:0/3661454453 >> 172.21.15.39:0/3661454453 conn(0x7fc9c80fe2c0 msgr2=0x7fc9c8100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 -- 172.21.15.39:0/3661454453 shutdown_connections 2024-09-21T01:04:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.696+0000 7fc9cda00640 1 -- 172.21.15.39:0/3661454453 wait complete. 2024-09-21T01:04:41.698 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:41 smithi039 ceph-mon[30415]: pgmap v424: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:41.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.697+0000 7fc9cda00640 1 Processor -- start 2024-09-21T01:04:41.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.697+0000 7fc9cda00640 1 -- start start 2024-09-21T01:04:41.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.698+0000 7fc9cda00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 0x7fc9c819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.698+0000 7fc9cda00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9c8105110 0x7fc9c819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.698+0000 7fc9cda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.698+0000 7fc9cda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c81138d0 con 0x7fc9c810dc20 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.698+0000 7fc9cda00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c8113750 con 0x7fc9c81047c0 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.698+0000 7fc9cda00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c8113a50 con 0x7fc9c8105110 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 0x7fc9c819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 0x7fc9c819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38910/0 (socket says 172.21.15.39:38910) 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7e00640 1 -- 172.21.15.39:0/274603491 learned_addr learned my addr 172.21.15.39:0/274603491 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7400640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9c8105110 0x7fc9c819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7e00640 1 -- 172.21.15.39:0/274603491 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9c8105110 msgr2=0x7fc9c819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7e00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9c8105110 0x7fc9c819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.699+0000 7fc9c7e00640 1 -- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 msgr2=0x7fc9c819fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.700+0000 7fc9cd000640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.700+0000 7fc9c7e00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c819fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.700+0000 7fc9c7e00640 1 -- 172.21.15.39:0/274603491 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c81a01b0 con 0x7fc9c81047c0 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.700+0000 7fc9cd000640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:04:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.701+0000 7fc9c7e00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 0x7fc9c819a240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800e9f0 tx=0x7fc9b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:41.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.701+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b800cd80 con 0x7fc9c81047c0 2024-09-21T01:04:41.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.701+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9b80108e0 con 0x7fc9c81047c0 2024-09-21T01:04:41.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.702+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c81a04a0 con 0x7fc9c81047c0 2024-09-21T01:04:41.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.702+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c81023e0 con 0x7fc9c81047c0 2024-09-21T01:04:41.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.702+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8018980 con 0x7fc9c81047c0 2024-09-21T01:04:41.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.703+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a80052c0 con 0x7fc9c81047c0 2024-09-21T01:04:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.706+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fc9b8010430 con 0x7fc9c81047c0 2024-09-21T01:04:41.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.707+0000 7fc9c4c00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9ac0b6780 0x7fc9ac0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:41.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.707+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc9b80e7ca0 con 0x7fc9c81047c0 2024-09-21T01:04:41.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.708+0000 7fc9c7400640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9ac0b6780 0x7fc9ac0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:41.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.709+0000 7fc9c7400640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9ac0b6780 0x7fc9ac0b8c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40095f0 tx=0x7fc9b4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:41.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:41.711+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc9b80ade10 con 0x7fc9c81047c0 2024-09-21T01:04:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:41 smithi150 ceph-mon[34945]: pgmap v424: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:41 smithi160 ceph-mon[34696]: pgmap v424: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-21T01:04:42.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.034+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9a8005880 con 0x7fc9c81047c0 2024-09-21T01:04:42.037 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:04:42.037 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:04:42.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.035+0000 7fc9c4c00640 1 -- 172.21.15.39:0/274603491 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7fc9b80b21f0 con 0x7fc9c81047c0 2024-09-21T01:04:42.037 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9ac0b6780 msgr2=0x7fc9ac0b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9ac0b6780 0x7fc9ac0b8c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40095f0 tx=0x7fc9b4005f90 comp rx=0 tx=0).stop 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 msgr2=0x7fc9c819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 0x7fc9c819a240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800e9f0 tx=0x7fc9b800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 shutdown_connections 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c810dc20 0x7fc9c819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9c8105110 0x7fc9c819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:42.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc9ac0b6780 0x7fc9ac0b8c90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:42.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 --2- 172.21.15.39:0/274603491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c81047c0 0x7fc9c819a240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:42.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 >> 172.21.15.39:0/274603491 conn(0x7fc9c80fe2c0 msgr2=0x7fc9c81076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:42.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.038+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 shutdown_connections 2024-09-21T01:04:42.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:42.039+0000 7fc9cda00640 1 -- 172.21.15.39:0/274603491 wait complete. 2024-09-21T01:04:42.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:42 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/274603491' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:42.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:42 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/274603491' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:42.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:42 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/274603491' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:04:43.108 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id fe47de4d --release --await' 2024-09-21T01:04:43.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id fe47de4d --release --await 2024-09-21T01:04:43.710 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:04:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:43 smithi150 ceph-mon[34945]: pgmap v425: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-09-21T01:04:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:43 smithi160 ceph-mon[34696]: pgmap v425: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-09-21T01:04:43.877 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:43 smithi039 ceph-mon[30415]: pgmap v425: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 -- 172.21.15.39:0/4012472199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c105110 msgr2=0x7f999c10d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 --2- 172.21.15.39:0/4012472199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c105110 0x7f999c10d660 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f998800fb10 tx=0x7f998802e570 comp rx=0 tx=0).stop 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 -- 172.21.15.39:0/4012472199 shutdown_connections 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 --2- 172.21.15.39:0/4012472199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c10dc20 0x7f999c110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 --2- 172.21.15.39:0/4012472199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c105110 0x7f999c10d660 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 --2- 172.21.15.39:0/4012472199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c1047c0 0x7f999c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.450+0000 7f99a0e00640 1 -- 172.21.15.39:0/4012472199 >> 172.21.15.39:0/4012472199 conn(0x7f999c0fe2c0 msgr2=0x7f999c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.451+0000 7f99a0e00640 1 -- 172.21.15.39:0/4012472199 shutdown_connections 2024-09-21T01:04:44.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.451+0000 7f99a0e00640 1 -- 172.21.15.39:0/4012472199 wait complete. 2024-09-21T01:04:44.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.452+0000 7f99a0e00640 1 Processor -- start 2024-09-21T01:04:44.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.452+0000 7f99a0e00640 1 -- start start 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f99a0e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c1047c0 0x7f999c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f99a0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 0x7f999c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f99a0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c10dc20 0x7f999c19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f99a0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1116c0 con 0x7f999c10dc20 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f99a0e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f999c111540 con 0x7f999c1047c0 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f99a0e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f999c111840 con 0x7f999c105110 2024-09-21T01:04:44.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f999aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 0x7f999c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f999b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c1047c0 0x7f999c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f999aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 0x7f999c198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53932/0 (socket says 172.21.15.39:53932) 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.453+0000 7f999aa00640 1 -- 172.21.15.39:0/420091564 learned_addr learned my addr 172.21.15.39:0/420091564 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999aa00640 1 -- 172.21.15.39:0/420091564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c1047c0 msgr2=0x7f999c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999aa00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c1047c0 0x7f999c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999aa00640 1 -- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c10dc20 msgr2=0x7f999c19d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999be00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c10dc20 0x7f999c19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999aa00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c10dc20 0x7f999c19d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999aa00640 1 -- 172.21.15.39:0/420091564 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c19e020 con 0x7f999c105110 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.454+0000 7f999be00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c10dc20 0x7f999c19d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.455+0000 7f999aa00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 0x7f999c198650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9988009230 tx=0x7f998800f430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.455+0000 7f9993e00640 1 -- 172.21.15.39:0/420091564 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988044070 con 0x7f999c105110 2024-09-21T01:04:44.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.455+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999c19e2b0 con 0x7f999c105110 2024-09-21T01:04:44.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.455+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999c1a4a70 con 0x7f999c105110 2024-09-21T01:04:44.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.455+0000 7f9993e00640 1 -- 172.21.15.39:0/420091564 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998800ee10 con 0x7f999c105110 2024-09-21T01:04:44.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.455+0000 7f9993e00640 1 -- 172.21.15.39:0/420091564 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988003be0 con 0x7f999c105110 2024-09-21T01:04:44.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.459+0000 7f9993e00640 1 -- 172.21.15.39:0/420091564 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f9988047070 con 0x7f999c105110 2024-09-21T01:04:44.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.460+0000 7f99a9048740 1 Processor -- start 2024-09-21T01:04:44.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.460+0000 7f99a9048740 1 -- start start 2024-09-21T01:04:44.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f99a9048740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448dc520 0x55bf44c8bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f99a9048740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448b3660 0x55bf44c8c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f99a9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c8c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f99a9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bf44bfa290 con 0x55bf448bb840 2024-09-21T01:04:44.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f99a9048740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55bf44be96c0 con 0x55bf448b3660 2024-09-21T01:04:44.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f99a9048740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55bf44bed4d0 con 0x55bf448dc520 2024-09-21T01:04:44.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f999aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c8c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f999aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c8c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43968/0 (socket says 172.21.15.39:43968) 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f999aa00640 1 -- 172.21.15.39:0/4244945423 learned_addr learned my addr 172.21.15.39:0/4244945423 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f999be00640 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448dc520 0x55bf44c8bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.461+0000 7f999b400640 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448b3660 0x55bf44c8c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f9993e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99800b6700 0x7f99800b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f9993e00640 1 -- 172.21.15.39:0/420091564 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f998810bf50 con 0x7f999c105110 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f999aa00640 1 -- 172.21.15.39:0/4244945423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448dc520 msgr2=0x55bf44c8bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f999aa00640 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448dc520 0x55bf44c8bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f999aa00640 1 -- 172.21.15.39:0/4244945423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448b3660 msgr2=0x55bf44c8c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f999aa00640 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448b3660 0x55bf44c8c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f999aa00640 1 -- 172.21.15.39:0/4244945423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf44b76460 con 0x55bf448bb840 2024-09-21T01:04:44.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.462+0000 7f999be00640 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448dc520 0x55bf44c8bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:04:44.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.463+0000 7f999be00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99800b6700 0x7f99800b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.463+0000 7f999aa00640 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c8c5d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9988109200 tx=0x7f998802e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e17893bbe2efd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:44.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.463+0000 7f9990c00640 1 -- 172.21.15.39:0/4244945423 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998804e070 con 0x55bf448bb840 2024-09-21T01:04:44.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.464+0000 7f9990c00640 1 -- 172.21.15.39:0/4244945423 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99880d14d0 con 0x55bf448bb840 2024-09-21T01:04:44.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.464+0000 7f9990c00640 1 -- 172.21.15.39:0/4244945423 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998804e070 con 0x55bf448bb840 2024-09-21T01:04:44.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 -- 172.21.15.39:0/4244945423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 msgr2=0x55bf44c8c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f999be00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99800b6700 0x7f99800b8c10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9994005fd0 tx=0x7f9994004ab0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:44.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c8c5d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9988109200 tx=0x7f998802e570 comp rx=0 tx=0).stop 2024-09-21T01:04:44.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 -- 172.21.15.39:0/4244945423 shutdown_connections 2024-09-21T01:04:44.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c8c5d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448b3660 0x55bf44c8c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 --2- 172.21.15.39:0/4244945423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448dc520 0x55bf44c8bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 -- 172.21.15.39:0/4244945423 >> 172.21.15.39:0/4244945423 conn(0x55bf448d0910 msgr2=0x55bf44c89a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 -- 172.21.15.39:0/4244945423 shutdown_connections 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.465+0000 7f99a9048740 1 -- 172.21.15.39:0/4244945423 wait complete. 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.466+0000 7f99a9048740 1 -- start start 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.466+0000 7f99a9048740 1 Processor -- start 2024-09-21T01:04:44.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c97740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c97cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44ca3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bf44a4abb0 con 0x55bf448dc520 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55bf44b846c0 con 0x55bf448bb840 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55bf44c9a0f0 con 0x55bf448b3660 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.467+0000 7f99a9048740 10 client.0 _ll_register_callbacks cb 0x55bf44b9f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c97740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c97740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53948/0 (socket says 172.21.15.39:53948) 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 -- 172.21.15.39:0/2150498792 learned_addr learned my addr 172.21.15.39:0/2150498792 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999be00640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44ca3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999aa00640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c97cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 -- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 msgr2=0x55bf44c97cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c97cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 -- 172.21.15.39:0/2150498792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 msgr2=0x55bf44ca3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44ca3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.468+0000 7f999b400640 1 -- 172.21.15.39:0/2150498792 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf44947b00 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.469+0000 7f999b400640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c97740 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f998c013710 tx=0x7f998c013bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d965f72404ff112 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.469+0000 7f9985600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.469+0000 7f9985600640 1 -- 172.21.15.39:0/2150498792 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c020060 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.469+0000 7f9985600640 1 -- 172.21.15.39:0/2150498792 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f998c011dd0 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.469+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99880d14d0 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.469+0000 7f9985600640 1 -- 172.21.15.39:0/2150498792 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f998c027050 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 10 client.25543 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bf449be960 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f9985600640 1 -- 172.21.15.39:0/2150498792 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99880d14d0 con 0x55bf448b3660 2024-09-21T01:04:44.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f9985600640 1 -- 172.21.15.39:0/2150498792 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f998c01a030 con 0x55bf448b3660 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 10 client.25543 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 10 client.25543 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 20 client.25543 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf4489aea0 0x55bf44ca94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 4 client.25543 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.470+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf44bfc1e0 con 0x55bf4489aea0 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.471+0000 7f999aa00640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf4489aea0 0x55bf44ca94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.472+0000 7f999aa00640 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf4489aea0 0x55bf44ca94f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c89a07704e57d4a1 server_cookie=10ea981befa69f19 in_seq=0 out_seq=0 2024-09-21T01:04:44.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.472+0000 7f9985600640 10 client.25543 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:44.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f9985600640 1 -- 172.21.15.39:0/2150498792 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bf44bfc1e0 con 0x55bf4489aea0 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f9985600640 10 client.25543 handle_command_reply: tid=0 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 client.25543 shutdown 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 msgr2=0x55bf44c97740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c97740 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f998c013710 tx=0x7f998c013bd0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 shutdown_connections 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf4489aea0 0x55bf44ca94f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44ca3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c97cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 --2- 172.21.15.39:0/2150498792 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c97740 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 >> 172.21.15.39:0/2150498792 conn(0x55bf448d0910 msgr2=0x55bf44c8ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f9985600640 0 client.25543 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.473+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 shutdown_connections 2024-09-21T01:04:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.474+0000 7f99a9048740 1 -- 172.21.15.39:0/2150498792 wait complete. 2024-09-21T01:04:44.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.474+0000 7f99a9048740 20 client.25543 trim_cache size 0 max 16384 2024-09-21T01:04:44.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.492+0000 7f99a9048740 1 Processor -- start 2024-09-21T01:04:44.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 -- start start 2024-09-21T01:04:44.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44ca3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c8ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44c8f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1b13e0 con 0x55bf448dc520 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1aecb0 con 0x55bf448bb840 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f99a9048740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1acb00 con 0x55bf448b3660 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f999b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44ca3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f999b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44ca3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53954/0 (socket says 172.21.15.39:53954) 2024-09-21T01:04:44.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f999b400640 1 -- 172.21.15.39:0/2309742815 learned_addr learned my addr 172.21.15.39:0/2309742815 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:44.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.493+0000 7f999aa00640 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c8ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.494+0000 7f999b400640 1 -- 172.21.15.39:0/2309742815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 msgr2=0x55bf44c8ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.494+0000 7f999b400640 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c8ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.494+0000 7f999b400640 1 -- 172.21.15.39:0/2309742815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 msgr2=0x55bf44c8f190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.498+0000 7f999be00640 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44c8f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.498+0000 7f999b400640 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44c8f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.498+0000 7f999b400640 1 -- 172.21.15.39:0/2309742815 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf44c95a40 con 0x55bf448b3660 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.498+0000 7f999be00640 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44c8f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.498+0000 7f999b400640 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44ca3070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f998c00e5c0 tx=0x7f998c005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56f41e6b8e232d5d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.498+0000 7f9987e00640 1 -- 172.21.15.39:0/2309742815 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c030070 con 0x55bf448b3660 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.499+0000 7f9987e00640 1 -- 172.21.15.39:0/2309742815 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998c011dd0 con 0x55bf448b3660 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.499+0000 7f9987e00640 1 -- 172.21.15.39:0/2309742815 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c004640 con 0x55bf448b3660 2024-09-21T01:04:44.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 -- 172.21.15.39:0/2309742815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 msgr2=0x55bf44ca3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44ca3070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f998c00e5c0 tx=0x7f998c005970 comp rx=0 tx=0).stop 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 -- 172.21.15.39:0/2309742815 shutdown_connections 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448dc520 0x55bf44c8f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448bb840 0x55bf44c8ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 --2- 172.21.15.39:0/2309742815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44ca3070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.500+0000 7f99a9048740 1 -- 172.21.15.39:0/2309742815 >> 172.21.15.39:0/2309742815 conn(0x55bf448d0910 msgr2=0x55bf44c95dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.501+0000 7f99a9048740 1 -- 172.21.15.39:0/2309742815 shutdown_connections 2024-09-21T01:04:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.501+0000 7f99a9048740 1 -- 172.21.15.39:0/2309742815 wait complete. 2024-09-21T01:04:44.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.501+0000 7f99a9048740 1 -- start start 2024-09-21T01:04:44.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.501+0000 7f99a9048740 1 Processor -- start 2024-09-21T01:04:44.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c9f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c9f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448dc520 0x55bf44c9a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1ad170 con 0x55bf448bb840 2024-09-21T01:04:44.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1ae5e0 con 0x55bf448dc520 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1ad840 con 0x55bf448b3660 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.502+0000 7f99a9048740 10 client.0 _ll_register_callbacks cb 0x7f998c0096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c9f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c9f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c9f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53958/0 (socket says 172.21.15.39:53958) 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 -- 172.21.15.39:0/2968050241 learned_addr learned my addr 172.21.15.39:0/2968050241 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999be00640 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448dc520 0x55bf44c9a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 -- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448dc520 msgr2=0x55bf44c9a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448dc520 0x55bf44c9a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 -- 172.21.15.39:0/2968050241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 msgr2=0x55bf44c9f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c9f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.503+0000 7f999b400640 1 -- 172.21.15.39:0/2968050241 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf44c9fbe0 con 0x55bf448b3660 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.504+0000 7f999b400640 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c9f0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f998c02e040 tx=0x7f998c009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b6409845bca4e05 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.504+0000 7f9985600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.504+0000 7f9985600640 1 -- 172.21.15.39:0/2968050241 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c030070 con 0x55bf448b3660 2024-09-21T01:04:44.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.504+0000 7f9985600640 1 -- 172.21.15.39:0/2968050241 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f998c005250 con 0x55bf448b3660 2024-09-21T01:04:44.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.504+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f998c011dd0 con 0x55bf448b3660 2024-09-21T01:04:44.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.505+0000 7f9985600640 1 -- 172.21.15.39:0/2968050241 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f998c020040 con 0x55bf448b3660 2024-09-21T01:04:44.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.505+0000 7f99a9048740 10 client.25549 fetch_fsmap learned FSMap version 27 2024-09-21T01:04:44.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.505+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bf44d141a0 con 0x55bf448b3660 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.505+0000 7f9985600640 1 -- 172.21.15.39:0/2968050241 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998c011dd0 con 0x55bf448b3660 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.506+0000 7f9985600640 1 -- 172.21.15.39:0/2968050241 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f998c01a040 con 0x55bf448b3660 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.506+0000 7f99a9048740 10 client.25549 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.506+0000 7f99a9048740 10 client.25549 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.506+0000 7f99a9048740 20 client.25549 populate_metadata read hostname 'smithi039' 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.506+0000 7f99a9048740 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf44870250 0x55bf44d167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.507+0000 7f99a9048740 4 client.25549 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fe47de4d", "release": true, "await": true}] 2024-09-21T01:04:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.507+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fe47de4d", "release": true, "await": true}) -- 0x55bf44bf7c90 con 0x55bf44870250 2024-09-21T01:04:44.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.507+0000 7f999aa00640 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf44870250 0x55bf44d167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:04:44.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.508+0000 7f999aa00640 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf44870250 0x55bf44d167f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5de45ecb00af331 server_cookie=2e35ce79e436cff7 in_seq=0 out_seq=0 2024-09-21T01:04:44.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.508+0000 7f9985600640 10 client.25549 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.510+0000 7f9985600640 1 -- 172.21.15.39:0/2968050241 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55bf44bf7c90 con 0x55bf44870250 2024-09-21T01:04:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.510+0000 7f9985600640 10 client.25549 handle_command_reply: tid=0 2024-09-21T01:04:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.510+0000 7f99a9048740 1 client.25549 shutdown 2024-09-21T01:04:44.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 msgr2=0x55bf44c9f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c9f0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f998c02e040 tx=0x7f998c009fd0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 shutdown_connections 2024-09-21T01:04:44.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf44870250 0x55bf44d167f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf448dc520 0x55bf44c9a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf448bb840 0x55bf44c9f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 --2- 172.21.15.39:0/2968050241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf448b3660 0x55bf44c9f0e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 >> 172.21.15.39:0/2968050241 conn(0x55bf448d0910 msgr2=0x55bf44c8ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:44.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.511+0000 7f9985600640 0 client.25549 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:04:44.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.512+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 shutdown_connections 2024-09-21T01:04:44.515 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:04:44.515 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 27, 2024-09-21T01:04:44.515 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:04:44.515 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 4, 2024-09-21T01:04:44.515 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "fe47de4d": { 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 4, 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 134.5, 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:04:44.516 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:04:44.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:44.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:44.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:44.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:44.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:04:44.517 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.512+0000 7f99a9048740 1 -- 172.21.15.39:0/2968050241 wait complete. 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.512+0000 7f99a9048740 20 client.25549 trim_cache size 0 max 16384 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.514+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99800b6700 msgr2=0x7f99800b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.514+0000 7f99a0e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99800b6700 0x7f99800b8c10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9994005fd0 tx=0x7f9994004ab0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.514+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 msgr2=0x7f999c198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.514+0000 7f99a0e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 0x7f999c198650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9988009230 tx=0x7f998800f430 comp rx=0 tx=0).stop 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.515+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 shutdown_connections 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.515+0000 7f99a0e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c10dc20 0x7f999c19d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.515+0000 7f99a0e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f999c105110 0x7f999c198650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.515+0000 7f99a0e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99800b6700 0x7f99800b8c10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.515+0000 7f99a0e00640 1 --2- 172.21.15.39:0/420091564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c1047c0 0x7f999c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:04:44.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.515+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 >> 172.21.15.39:0/420091564 conn(0x7f999c0fe2c0 msgr2=0x7f999c0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:04:44.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.516+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 shutdown_connections 2024-09-21T01:04:44.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:04:44.516+0000 7f99a0e00640 1 -- 172.21.15.39:0/420091564 wait complete. 2024-09-21T01:04:45.533 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fe47de4d', total seconds elapsed: 19.6 2024-09-21T01:04:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:45 smithi150 ceph-mon[34945]: pgmap v426: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:45 smithi160 ceph-mon[34696]: pgmap v426: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:45 smithi039 ceph-mon[30415]: pgmap v426: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:47 smithi150 ceph-mon[34945]: pgmap v427: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:47.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:47 smithi160 ceph-mon[34696]: pgmap v427: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:47 smithi039 ceph-mon[30415]: pgmap v427: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:49 smithi150 ceph-mon[34945]: pgmap v428: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:49 smithi160 ceph-mon[34696]: pgmap v428: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:49 smithi039 ceph-mon[30415]: pgmap v428: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:51 smithi150 ceph-mon[34945]: pgmap v429: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:51.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:51 smithi160 ceph-mon[34696]: pgmap v429: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:51 smithi039 ceph-mon[30415]: pgmap v429: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:04:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:53 smithi150 ceph-mon[34945]: pgmap v430: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:04:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:53 smithi160 ceph-mon[34696]: pgmap v430: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:53.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:04:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:53 smithi039 ceph-mon[30415]: pgmap v430: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:04:54.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:04:54.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:04:54.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:54 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:04:54.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:04:54.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:04:54.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:54 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:04:54.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:04:54.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:04:54.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:54 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:04:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:55 smithi150 ceph-mon[34945]: pgmap v431: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:55 smithi160 ceph-mon[34696]: pgmap v431: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:55 smithi039 ceph-mon[30415]: pgmap v431: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:04:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:57 smithi150 ceph-mon[34945]: pgmap v432: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:57 smithi160 ceph-mon[34696]: pgmap v432: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:57 smithi039 ceph-mon[30415]: pgmap v432: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:04:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:04:59 smithi160 ceph-mon[34696]: pgmap v433: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:04:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:04:59 smithi039 ceph-mon[30415]: pgmap v433: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:05:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:04:59 smithi150 ceph-mon[34945]: pgmap v433: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-21T01:05:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:01 smithi160 ceph-mon[34696]: pgmap v434: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:05:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:01 smithi039 ceph-mon[30415]: pgmap v434: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:05:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:01 smithi150 ceph-mon[34945]: pgmap v434: 129 pgs: 129 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-21T01:05:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:03 smithi160 ceph-mon[34696]: pgmap v435: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-09-21T01:05:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:03 smithi039 ceph-mon[30415]: pgmap v435: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-09-21T01:05:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:03 smithi150 ceph-mon[34945]: pgmap v435: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-09-21T01:05:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:05 smithi160 ceph-mon[34696]: pgmap v436: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-09-21T01:05:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:05 smithi039 ceph-mon[30415]: pgmap v436: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-09-21T01:05:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:05 smithi150 ceph-mon[34945]: pgmap v436: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-09-21T01:05:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:07 smithi160 ceph-mon[34696]: pgmap v437: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:07 smithi039 ceph-mon[30415]: pgmap v437: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:07 smithi150 ceph-mon[34945]: pgmap v437: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:09 smithi160 ceph-mon[34696]: pgmap v438: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:09 smithi039 ceph-mon[30415]: pgmap v438: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:09 smithi150 ceph-mon[34945]: pgmap v438: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:11 smithi039 ceph-mon[30415]: pgmap v439: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:11 smithi150 ceph-mon[34945]: pgmap v439: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:11 smithi160 ceph-mon[34696]: pgmap v439: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-09-21T01:05:12.462 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.0 2024-09-21T01:05:12.463 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:05:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:05:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 -- 172.21.15.39:0/1795967513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c78102f40 msgr2=0x7f3c7810fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 --2- 172.21.15.39:0/1795967513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c78102f40 0x7f3c7810fc60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009900 tx=0x7f3c6002a480 comp rx=0 tx=0).stop 2024-09-21T01:05:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 -- 172.21.15.39:0/1795967513 shutdown_connections 2024-09-21T01:05:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 --2- 172.21.15.39:0/1795967513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c781101a0 0x7f3c781125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 --2- 172.21.15.39:0/1795967513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c78102f40 0x7f3c7810fc60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 --2- 172.21.15.39:0/1795967513 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c781025f0 0x7f3c78102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 -- 172.21.15.39:0/1795967513 >> 172.21.15.39:0/1795967513 conn(0x7f3c78079740 msgr2=0x7f3c78079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 -- 172.21.15.39:0/1795967513 shutdown_connections 2024-09-21T01:05:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.662+0000 7f3c7f200640 1 -- 172.21.15.39:0/1795967513 wait complete. 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.663+0000 7f3c7f200640 1 Processor -- start 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 -- start start 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 0x7f3c7819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c78102f40 0x7f3c7819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c781101a0 0x7f3c781a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78115c90 con 0x7f3c781025f0 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78115b10 con 0x7f3c781101a0 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c7f200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78115e10 con 0x7f3c78102f40 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c77e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 0x7f3c7819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:13.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.664+0000 7f3c6fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c78102f40 0x7f3c7819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 0x7f3c7819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37242/0 (socket says 172.21.15.39:37242) 2024-09-21T01:05:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 -- 172.21.15.39:0/3771248237 learned_addr learned my addr 172.21.15.39:0/3771248237 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c6fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c78102f40 0x7f3c7819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:57070/0 (socket says 172.21.15.39:57070) 2024-09-21T01:05:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c7d200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c781101a0 0x7f3c781a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 -- 172.21.15.39:0/3771248237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c78102f40 msgr2=0x7f3c7819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c78102f40 0x7f3c7819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 -- 172.21.15.39:0/3771248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c781101a0 msgr2=0x7f3c781a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c781101a0 0x7f3c781a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 -- 172.21.15.39:0/3771248237 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c781a25f0 con 0x7f3c781025f0 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.665+0000 7f3c77e00640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 0x7f3c7819c680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800e9f0 tx=0x7f3c6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.666+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6800cd80 con 0x7f3c781025f0 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.666+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c781a28e0 con 0x7f3c781025f0 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.666+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c680108e0 con 0x7f3c781025f0 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.666+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68018980 con 0x7f3c781025f0 2024-09-21T01:05:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.666+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c781a9040 con 0x7f3c781025f0 2024-09-21T01:05:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.667+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7806c240 con 0x7f3c781025f0 2024-09-21T01:05:13.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.670+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3c68010430 con 0x7f3c781025f0 2024-09-21T01:05:13.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.671+0000 7f3c75600640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3c5c0b6680 0x7f3c5c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:13.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.671+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3c680e7e30 con 0x7f3c781025f0 2024-09-21T01:05:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.671+0000 7f3c6fe00640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3c5c0b6680 0x7f3c5c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.672+0000 7f3c6fe00640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3c5c0b6680 0x7f3c5c0b8b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3c60031440 tx=0x7f3c60009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:13.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.676+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3c680b2380 con 0x7f3c781025f0 2024-09-21T01:05:13.752 INFO:tasks.workunit.client.0.smithi039.stderr:Updating files: 59% (7525/12697) Updating files: 60% (7619/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 93% (11932/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-21T01:05:13.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:13 smithi039 ceph-mon[30415]: pgmap v440: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-09-21T01:05:13.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.927+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c7819d7b0 con 0x7f3c781025f0 2024-09-21T01:05:13.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.928+0000 7f3c75600640 1 -- 172.21.15.39:0/3771248237 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7f3c680b72c0 con 0x7f3c781025f0 2024-09-21T01:05:13.929 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:05:13.929 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:05:13.929 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:05:13.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3c5c0b6680 msgr2=0x7f3c5c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3c5c0b6680 0x7f3c5c0b8b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3c60031440 tx=0x7f3c60009560 comp rx=0 tx=0).stop 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 msgr2=0x7f3c7819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 0x7f3c7819c680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800e9f0 tx=0x7f3c6800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 shutdown_connections 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3c5c0b6680 0x7f3c5c0b8b90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c781101a0 0x7f3c781a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c78102f40 0x7f3c7819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 --2- 172.21.15.39:0/3771248237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c781025f0 0x7f3c7819c680 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 >> 172.21.15.39:0/3771248237 conn(0x7f3c78079740 msgr2=0x7f3c78109d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 shutdown_connections 2024-09-21T01:05:13.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:13.931+0000 7f3c7f200640 1 -- 172.21.15.39:0/3771248237 wait complete. 2024-09-21T01:05:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:13 smithi150 ceph-mon[34945]: pgmap v440: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-09-21T01:05:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:13 smithi160 ceph-mon[34696]: pgmap v440: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-09-21T01:05:14.836 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to '1843221895adccfb3b692060251dd6a0183bc410'. 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-09-21T01:05:14.837 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.838 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T01:05:14.838 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-21T01:05:14.838 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at 1843221895a Merge PR #56776 into wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:05:14.851 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T01:05:14.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:14 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3771248237' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:05:14.948 INFO:tasks.workunit.client.0.smithi039.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T01:05:14.954 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T01:05:14.954 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T01:05:14.980 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 137.0 --await-for 45.0' 2024-09-21T01:05:14.980 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 137.0 --await-for 45.0 2024-09-21T01:05:15.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:14 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3771248237' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:05:15.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:14 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3771248237' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:05:15.391 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:05:15.883 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:15 smithi039 ceph-mon[30415]: pgmap v441: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-21T01:05:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:15 smithi150 ceph-mon[34945]: pgmap v441: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-21T01:05:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:15 smithi160 ceph-mon[34696]: pgmap v441: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-21T01:05:16.237 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T01:05:16.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 -- 172.21.15.39:0/1056316935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3940076100 msgr2=0x7f394010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 --2- 172.21.15.39:0/1056316935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3940076100 0x7f394010fba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f392c009900 tx=0x7f392c02a410 comp rx=0 tx=0).stop 2024-09-21T01:05:16.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 -- 172.21.15.39:0/1056316935 shutdown_connections 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 --2- 172.21.15.39:0/1056316935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39401100e0 0x7f3940112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 --2- 172.21.15.39:0/1056316935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3940076100 0x7f394010fba0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 --2- 172.21.15.39:0/1056316935 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39400757b0 0x7f3940075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 -- 172.21.15.39:0/1056316935 >> 172.21.15.39:0/1056316935 conn(0x7f39400ff5d0 msgr2=0x7f3940101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 -- 172.21.15.39:0/1056316935 shutdown_connections 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.257+0000 7f3948400640 1 -- 172.21.15.39:0/1056316935 wait complete. 2024-09-21T01:05:16.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.258+0000 7f3948400640 1 Processor -- start 2024-09-21T01:05:16.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 -- start start 2024-09-21T01:05:16.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39400757b0 0x7f394019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3940076100 0x7f394019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39401100e0 0x7f39401a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3940115c90 con 0x7f39400757b0 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3940115b10 con 0x7f39401100e0 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.259+0000 7f3948400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3940115e10 con 0x7f3940076100 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39400757b0 0x7f394019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39400757b0 0x7f394019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37266/0 (socket says 172.21.15.39:37266) 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945a00640 1 -- 172.21.15.39:0/499758877 learned_addr learned my addr 172.21.15.39:0/499758877 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:16.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945000640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3940076100 0x7f394019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3946400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39401100e0 0x7f39401a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945000640 1 -- 172.21.15.39:0/499758877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39401100e0 msgr2=0x7f39401a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945000640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39401100e0 0x7f39401a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945000640 1 -- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39400757b0 msgr2=0x7f394019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945000640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39400757b0 0x7f394019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.260+0000 7f3945000640 1 -- 172.21.15.39:0/499758877 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39401a2630 con 0x7f3940076100 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f3946400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39401100e0 0x7f39401a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f3945000640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3940076100 0x7f394019cc60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f392c02a930 tx=0x7f392c031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f393e000640 1 -- 172.21.15.39:0/499758877 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c03f070 con 0x7f3940076100 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f393e000640 1 -- 172.21.15.39:0/499758877 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c004490 con 0x7f3940076100 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39401a28c0 con 0x7f3940076100 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f393e000640 1 -- 172.21.15.39:0/499758877 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c035070 con 0x7f3940076100 2024-09-21T01:05:16.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.261+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39401a9080 con 0x7f3940076100 2024-09-21T01:05:16.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.266+0000 7f393e000640 1 -- 172.21.15.39:0/499758877 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f392c03b460 con 0x7f3940076100 2024-09-21T01:05:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 Processor -- start 2024-09-21T01:05:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 -- start start 2024-09-21T01:05:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9de8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556af9d5dca0 con 0x556af9a1fb60 2024-09-21T01:05:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556af9d4d0d0 con 0x556af9a40840 2024-09-21T01:05:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.267+0000 7f394f026740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x556af9d50ee0 con 0x556af9a17980 2024-09-21T01:05:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f3946400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f3946400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:59964/0 (socket says 172.21.15.39:59964) 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f3946400640 1 -- 172.21.15.39:0/4268993800 learned_addr learned my addr 172.21.15.39:0/4268993800 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f393e000640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39280b6680 0x7f39280b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f3945a00640 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9de8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f3946400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39280b6680 0x7f39280b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.268+0000 7f3945000640 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945a00640 1 -- 172.21.15.39:0/4268993800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 msgr2=0x556af9de8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945a00640 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945a00640 1 -- 172.21.15.39:0/4268993800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 msgr2=0x556af9de8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945a00640 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f393e000640 1 -- 172.21.15.39:0/499758877 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f392c107f40 con 0x7f3940076100 2024-09-21T01:05:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945a00640 1 -- 172.21.15.39:0/4268993800 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556af9cff400 con 0x556af9a17980 2024-09-21T01:05:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945000640 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3945a00640 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9de8890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3934009cf0 tx=0x7f3934011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972a8dc8f8dd3db4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3946400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39280b6680 0x7f39280b8b90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3938005c40 tx=0x7f3938004ab0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.269+0000 7f3932a00640 1 -- 172.21.15.39:0/4268993800 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934005220 con 0x556af9a17980 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.270+0000 7f3932a00640 1 -- 172.21.15.39:0/4268993800 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3934004c50 con 0x556af9a17980 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.270+0000 7f3932a00640 1 -- 172.21.15.39:0/4268993800 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39340197f0 con 0x556af9a17980 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.270+0000 7f394f026740 1 -- 172.21.15.39:0/4268993800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 msgr2=0x556af9de8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.270+0000 7f394f026740 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9de8890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3934009cf0 tx=0x7f3934011810 comp rx=0 tx=0).stop 2024-09-21T01:05:16.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 -- 172.21.15.39:0/4268993800 shutdown_connections 2024-09-21T01:05:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9de8890 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 --2- 172.21.15.39:0/4268993800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 -- 172.21.15.39:0/4268993800 >> 172.21.15.39:0/4268993800 conn(0x556af9a34990 msgr2=0x556af9de6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 -- 172.21.15.39:0/4268993800 shutdown_connections 2024-09-21T01:05:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 -- 172.21.15.39:0/4268993800 wait complete. 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.271+0000 7f394f026740 1 -- start start 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 Processor -- start 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 0x556af9df3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a1fb60 0x556af9df3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9dffe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556af9bae7a0 con 0x556af9a17980 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556af9cf2a10 con 0x556af9a40840 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x556af9df6b30 con 0x556af9a1fb60 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.272+0000 7f394f026740 10 client.0 _ll_register_callbacks cb 0x556af9d02be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3945a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 0x556af9df3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3945a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 0x556af9df3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37294/0 (socket says 172.21.15.39:37294) 2024-09-21T01:05:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3945a00640 1 -- 172.21.15.39:0/3024434341 learned_addr learned my addr 172.21.15.39:0/3024434341 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:16.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3946400640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9dffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3945000640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a1fb60 0x556af9df3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3946400640 1 -- 172.21.15.39:0/3024434341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a1fb60 msgr2=0x556af9df3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3946400640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a1fb60 0x556af9df3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3946400640 1 -- 172.21.15.39:0/3024434341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 msgr2=0x556af9df3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3946400640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 0x556af9df3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3946400640 1 -- 172.21.15.39:0/3024434341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556af9abab90 con 0x556af9a40840 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3945000640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a1fb60 0x556af9df3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.273+0000 7f3945a00640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 0x556af9df3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:16.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.274+0000 7f3946400640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9dffe50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3938009fd0 tx=0x7f3938004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cc47d6f1cfe18f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.274+0000 7f3927400640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.274+0000 7f3927400640 1 -- 172.21.15.39:0/3024434341 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938004d20 con 0x556af9a40840 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.274+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3934004c50 con 0x556af9a40840 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.274+0000 7f3927400640 1 -- 172.21.15.39:0/3024434341 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f393801e620 con 0x556af9a40840 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.275+0000 7f3927400640 1 -- 172.21.15.39:0/3024434341 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f393801e820 con 0x556af9a40840 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.275+0000 7f394f026740 10 client.25577 fetch_fsmap learned FSMap version 27 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.275+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556af9e02800 con 0x556af9a40840 2024-09-21T01:05:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.275+0000 7f3927400640 1 -- 172.21.15.39:0/3024434341 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3934004c50 con 0x556af9a40840 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.275+0000 7f3927400640 1 -- 172.21.15.39:0/3024434341 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3938027030 con 0x556af9a40840 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f394f026740 10 client.25577 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f394f026740 10 client.25577 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f394f026740 20 client.25577 populate_metadata read hostname 'smithi039' 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f394f026740 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99ff1c0 0x556af9e05e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f394f026740 4 client.25577 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556af9d5fbf0 con 0x556af99ff1c0 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.276+0000 7f3945a00640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99ff1c0 0x556af9e05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.277+0000 7f3945a00640 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99ff1c0 0x556af9e05e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55d11beecfbebeb6 server_cookie=2f2e5471d89b72a6 in_seq=0 out_seq=0 2024-09-21T01:05:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.277+0000 7f3927400640 10 client.25577 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.277+0000 7f3927400640 1 -- 172.21.15.39:0/3024434341 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556af9d5fbf0 con 0x556af99ff1c0 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.277+0000 7f3927400640 10 client.25577 handle_command_reply: tid=0 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.277+0000 7f394f026740 1 client.25577 shutdown 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 msgr2=0x556af9dffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9dffe50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3938009fd0 tx=0x7f3938004a40 comp rx=0 tx=0).stop 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 shutdown_connections 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99ff1c0 0x556af9e05e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9dffe50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f3927400640 0 client.25577 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a1fb60 0x556af9df3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 --2- 172.21.15.39:0/3024434341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a17980 0x556af9df3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 >> 172.21.15.39:0/3024434341 conn(0x556af9a34990 msgr2=0x556af9de49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 shutdown_connections 2024-09-21T01:05:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 1 -- 172.21.15.39:0/3024434341 wait complete. 2024-09-21T01:05:16.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.278+0000 7f394f026740 20 client.25577 trim_cache size 0 max 16384 2024-09-21T01:05:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.296+0000 7f394f026740 1 Processor -- start 2024-09-21T01:05:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.296+0000 7f394f026740 1 -- start start 2024-09-21T01:05:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f394f026740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9def1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f394f026740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f394f026740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f394f026740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39401b5900 con 0x556af9a1fb60 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f394f026740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39401b31d0 con 0x556af9a40840 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f394f026740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39401b1020 con 0x556af9a17980 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f3945000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f3945000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37308/0 (socket says 172.21.15.39:37308) 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f3945000640 1 -- 172.21.15.39:0/762245380 learned_addr learned my addr 172.21.15.39:0/762245380 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f3945a00640 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9def1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.297+0000 7f3946400640 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3945000640 1 -- 172.21.15.39:0/762245380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 msgr2=0x556af9def1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3945000640 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9def1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3945000640 1 -- 172.21.15.39:0/762245380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 msgr2=0x556af9de9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3945000640 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3945000640 1 -- 172.21.15.39:0/762245380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556af9de1c50 con 0x556af9a1fb60 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3946400640 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3945000640 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de5360 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f392c0ca030 tx=0x7f392c099610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8126932d1278e207 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.298+0000 7f3932000640 1 -- 172.21.15.39:0/762245380 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c048040 con 0x556af9a1fb60 2024-09-21T01:05:16.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.299+0000 7f3932000640 1 -- 172.21.15.39:0/762245380 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c0c7040 con 0x556af9a1fb60 2024-09-21T01:05:16.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.299+0000 7f3932000640 1 -- 172.21.15.39:0/762245380 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c091e20 con 0x556af9a1fb60 2024-09-21T01:05:16.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 -- 172.21.15.39:0/762245380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 msgr2=0x556af9de5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de5360 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f392c0ca030 tx=0x7f392c099610 comp rx=0 tx=0).stop 2024-09-21T01:05:16.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 -- 172.21.15.39:0/762245380 shutdown_connections 2024-09-21T01:05:16.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a40840 0x556af9de9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a1fb60 0x556af9de5360 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 --2- 172.21.15.39:0/762245380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9def1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 -- 172.21.15.39:0/762245380 >> 172.21.15.39:0/762245380 conn(0x556af9a34990 msgr2=0x556af9de9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 -- 172.21.15.39:0/762245380 shutdown_connections 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.300+0000 7f394f026740 1 -- 172.21.15.39:0/762245380 wait complete. 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 -- start start 2024-09-21T01:05:16.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 Processor -- start 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9df7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a1fb60 0x556af9df81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a40840 0x556af9df8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39401b1690 con 0x556af9a40840 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39401b2b00 con 0x556af9a1fb60 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.301+0000 7f394f026740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39401b1d60 con 0x556af9a17980 2024-09-21T01:05:16.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f394f026740 10 client.0 _ll_register_callbacks cb 0x7f39380137a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:05:16.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3946400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a40840 0x556af9df8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9df7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a1fb60 0x556af9df81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3946400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a40840 0x556af9df8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37310/0 (socket says 172.21.15.39:37310) 2024-09-21T01:05:16.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9df7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:57124/0 (socket says 172.21.15.39:57124) 2024-09-21T01:05:16.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3946400640 1 -- 172.21.15.39:0/1614446278 learned_addr learned my addr 172.21.15.39:0/1614446278 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:16.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 -- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a1fb60 msgr2=0x556af9df81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a1fb60 0x556af9df81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 -- 172.21.15.39:0/1614446278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a40840 msgr2=0x556af9df8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:16.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a40840 0x556af9df8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.302+0000 7f3945a00640 1 -- 172.21.15.39:0/1614446278 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556af9e6f7e0 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.303+0000 7f3945000640 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a1fb60 0x556af9df81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.303+0000 7f3945a00640 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9df7cb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3934009fa0 tx=0x7f39340097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfea187d6436b289 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.303+0000 7f3927400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.303+0000 7f3927400640 1 -- 172.21.15.39:0/1614446278 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934004e30 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.303+0000 7f3927400640 1 -- 172.21.15.39:0/1614446278 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3934004fd0 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.303+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f392c0c7040 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f3927400640 1 -- 172.21.15.39:0/1614446278 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f39340051d0 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 10 client.25573 fetch_fsmap learned FSMap version 27 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556af9e70960 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f3927400640 1 -- 172.21.15.39:0/1614446278 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c0c7040 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f3927400640 1 -- 172.21.15.39:0/1614446278 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3934034420 con 0x556af9a17980 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 10 client.25573 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 10 client.25573 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:05:16.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 20 client.25573 populate_metadata read hostname 'smithi039' 2024-09-21T01:05:16.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99d4570 0x556af9e72f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:16.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 4 client.25573 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.0, "await_for": 45.0}] 2024-09-21T01:05:16.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.304+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.0, "await_for": 45.0}) -- 0x556af9d5b6a0 con 0x556af99d4570 2024-09-21T01:05:16.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.305+0000 7f3945000640 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99d4570 0x556af9e72f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:16.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.305+0000 7f3927400640 10 client.25573 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:16.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.305+0000 7f3945000640 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99d4570 0x556af9e72f70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc82429645bd639f server_cookie=230b07a2c61903b7 in_seq=0 out_seq=0 2024-09-21T01:05:16.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f3927400640 1 -- 172.21.15.39:0/1614446278 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x556af9d5b6a0 con 0x556af99d4570 2024-09-21T01:05:16.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f3927400640 10 client.25573 handle_command_reply: tid=0 2024-09-21T01:05:16.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 client.25573 shutdown 2024-09-21T01:05:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 msgr2=0x556af9df7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9df7cb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3934009fa0 tx=0x7f39340097b0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 shutdown_connections 2024-09-21T01:05:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x556af99d4570 0x556af9e72f70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556af9a40840 0x556af9df8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556af9a1fb60 0x556af9df81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 --2- 172.21.15.39:0/1614446278 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x556af9a17980 0x556af9df7cb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.316+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 >> 172.21.15.39:0/1614446278 conn(0x556af9a34990 msgr2=0x556af9de5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.317+0000 7f3927400640 0 client.25573 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.317+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 shutdown_connections 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.317+0000 7f394f026740 1 -- 172.21.15.39:0/1614446278 wait complete. 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.317+0000 7f394f026740 20 client.25573 trim_cache size 0 max 16384 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:05:16.319 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 27, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 6, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ce59dea0": { 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 6, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 137.0, 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:05:16.320 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:16.321 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.318+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39280b6680 msgr2=0x7f39280b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.318+0000 7f3948400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39280b6680 0x7f39280b8b90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3938005c40 tx=0x7f3938004ab0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.318+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3940076100 msgr2=0x7f394019cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.318+0000 7f3948400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3940076100 0x7f394019cc60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f392c02a930 tx=0x7f392c031a20 comp rx=0 tx=0).stop 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 shutdown_connections 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39280b6680 0x7f39280b8b90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39401100e0 0x7f39401a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3940076100 0x7f394019cc60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 --2- 172.21.15.39:0/499758877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39400757b0 0x7f394019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 >> 172.21.15.39:0/499758877 conn(0x7f39400ff5d0 msgr2=0x7f3940109c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:16.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 shutdown_connections 2024-09-21T01:05:16.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:16.319+0000 7f3948400640 1 -- 172.21.15.39:0/499758877 wait complete. 2024-09-21T01:05:16.345 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T01:05:16.416 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T01:05:16.420 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T01:05:16.420 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T01:05:16.486 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T01:05:16.494 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:05:16.494 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T01:05:16.530 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-09-21T01:05:16.532 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-21T01:05:16.532 DEBUG:teuthology.orchestra.run.smithi039:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1843221895adccfb3b692060251dd6a0183bc410 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-09-21T01:05:17.200 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ce59dea0', quiesce took 4.7 seconds. Will release after: 12.3 2024-09-21T01:05:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:17 smithi039 ceph-mon[30415]: pgmap v442: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T01:05:17.909 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:17 smithi160 ceph-mon[34696]: pgmap v442: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T01:05:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:17 smithi150 ceph-mon[34945]: pgmap v442: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T01:05:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:19 smithi039 ceph-mon[30415]: pgmap v443: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:05:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:19 smithi150 ceph-mon[34945]: pgmap v443: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:05:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:19 smithi160 ceph-mon[34696]: pgmap v443: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:05:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:21 smithi039 ceph-mon[30415]: pgmap v444: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:05:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:21 smithi150 ceph-mon[34945]: pgmap v444: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:05:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:21 smithi160 ceph-mon[34696]: pgmap v444: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-21T01:05:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:23 smithi039 ceph-mon[30415]: pgmap v445: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:23 smithi150 ceph-mon[34945]: pgmap v445: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:23 smithi160 ceph-mon[34696]: pgmap v445: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:26.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:25 smithi039 ceph-mon[30415]: pgmap v446: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:05:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:25 smithi150 ceph-mon[34945]: pgmap v446: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:05:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:25 smithi160 ceph-mon[34696]: pgmap v446: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-09-21T01:05:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:27 smithi150 ceph-mon[34945]: pgmap v447: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:27 smithi160 ceph-mon[34696]: pgmap v447: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:27 smithi039 ceph-mon[30415]: pgmap v447: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:29.501 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:05:29.897 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:05:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:29 smithi039 ceph-mon[30415]: pgmap v448: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:29 smithi150 ceph-mon[34945]: pgmap v448: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:30.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:29 smithi160 ceph-mon[34696]: pgmap v448: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 -- 172.21.15.39:0/3377678289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810c040 msgr2=0x7f5b08112870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 --2- 172.21.15.39:0/3377678289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810c040 0x7f5b08112870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009ca0 tx=0x7f5b0002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 -- 172.21.15.39:0/3377678289 shutdown_connections 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 --2- 172.21.15.39:0/3377678289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810c040 0x7f5b08112870 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 --2- 172.21.15.39:0/3377678289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0810b6f0 0x7f5b0810bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 --2- 172.21.15.39:0/3377678289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b08105db0 0x7f5b081061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 -- 172.21.15.39:0/3377678289 >> 172.21.15.39:0/3377678289 conn(0x7f5b081017e0 msgr2=0x7f5b08103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:30.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 -- 172.21.15.39:0/3377678289 shutdown_connections 2024-09-21T01:05:30.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.572+0000 7f5b0d000640 1 -- 172.21.15.39:0/3377678289 wait complete. 2024-09-21T01:05:30.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 Processor -- start 2024-09-21T01:05:30.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 -- start start 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b08105db0 0x7f5b0819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810b6f0 0x7f5b0819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b0810c040 0x7f5b081a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08115d50 con 0x7f5b0810b6f0 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08115bd0 con 0x7f5b08105db0 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.574+0000 7f5b0d000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08115ed0 con 0x7f5b0810c040 2024-09-21T01:05:30.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5affe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810b6f0 0x7f5b0819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:30.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b05e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b08105db0 0x7f5b0819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:30.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b06800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b0810c040 0x7f5b081a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:30.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5affe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810b6f0 0x7f5b0819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43048/0 (socket says 172.21.15.39:43048) 2024-09-21T01:05:30.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5affe00640 1 -- 172.21.15.39:0/2737956477 learned_addr learned my addr 172.21.15.39:0/2737956477 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b06800640 1 -- 172.21.15.39:0/2737956477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b08105db0 msgr2=0x7f5b0819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b06800640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b08105db0 0x7f5b0819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b06800640 1 -- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810b6f0 msgr2=0x7f5b0819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b06800640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810b6f0 0x7f5b0819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.575+0000 7f5b06800640 1 -- 172.21.15.39:0/2737956477 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b081a2690 con 0x7f5b0810c040 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.576+0000 7f5b06800640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b0810c040 0x7f5b081a1f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005e30 tx=0x7f5b000317c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.576+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00003b70 con 0x7f5b0810c040 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.576+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b00003d10 con 0x7f5b0810c040 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.576+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b081a2920 con 0x7f5b0810c040 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.576+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0002f970 con 0x7f5b0810c040 2024-09-21T01:05:30.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.576+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b081a90e0 con 0x7f5b0810c040 2024-09-21T01:05:30.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.577+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b081061c0 con 0x7f5b0810c040 2024-09-21T01:05:30.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.580+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5b00004b40 con 0x7f5b0810c040 2024-09-21T01:05:30.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.581+0000 7f5afea00640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5aec0b6680 0x7f5aec0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:30.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.582+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5b00108c90 con 0x7f5b0810c040 2024-09-21T01:05:30.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.582+0000 7f5b05e00640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5aec0b6680 0x7f5aec0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:30.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.582+0000 7f5b05e00640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5aec0b6680 0x7f5aec0b8b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009bd0 tx=0x7f5af8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:30.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.583+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5b000cee00 con 0x7f5b0810c040 2024-09-21T01:05:30.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.821+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b0819d850 con 0x7f5b0810c040 2024-09-21T01:05:30.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.822+0000 7f5afea00640 1 -- 172.21.15.39:0/2737956477 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11911 (secure 0 0 0) 0x7f5b0819d850 con 0x7f5b0810c040 2024-09-21T01:05:30.824 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:05:30.824 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"btime":"2024-09-21T01:03:45:020907+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:03:44.185282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:05:30.824 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5aec0b6680 msgr2=0x7f5aec0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5aec0b6680 0x7f5aec0b8b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009bd0 tx=0x7f5af8005f90 comp rx=0 tx=0).stop 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b0810c040 msgr2=0x7f5b081a1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b0810c040 0x7f5b081a1f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005e30 tx=0x7f5b000317c0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 shutdown_connections 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5b0810c040 0x7f5b081a1f60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b0810b6f0 0x7f5b0819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5aec0b6680 0x7f5aec0b8b90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 --2- 172.21.15.39:0/2737956477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b08105db0 0x7f5b0819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 >> 172.21.15.39:0/2737956477 conn(0x7f5b081017e0 msgr2=0x7f5b08108c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.825+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 shutdown_connections 2024-09-21T01:05:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:30.826+0000 7f5b0d000640 1 -- 172.21.15.39:0/2737956477 wait complete. 2024-09-21T01:05:31.812 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id ce59dea0 --release --await' 2024-09-21T01:05:31.812 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id ce59dea0 --release --await 2024-09-21T01:05:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:31 smithi150 ceph-mon[34945]: pgmap v449: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:05:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:31 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2737956477' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:05:32.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:31 smithi160 ceph-mon[34696]: pgmap v449: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:05:32.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:31 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2737956477' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:05:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:31 smithi039 ceph-mon[30415]: pgmap v449: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:05:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:31 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2737956477' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:05:32.191 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:05:32.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.899+0000 7f547c000640 1 -- 172.21.15.39:0/1284787949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 msgr2=0x7f54741098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.899+0000 7f547c000640 1 --2- 172.21.15.39:0/1284787949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f54741098c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5460009900 tx=0x7f546002a410 comp rx=0 tx=0).stop 2024-09-21T01:05:32.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 -- 172.21.15.39:0/1284787949 shutdown_connections 2024-09-21T01:05:32.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 --2- 172.21.15.39:0/1284787949 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474109e00 0x7f5474110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 --2- 172.21.15.39:0/1284787949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f54741098c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 --2- 172.21.15.39:0/1284787949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5474103b70 0x7f5474103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 -- 172.21.15.39:0/1284787949 >> 172.21.15.39:0/1284787949 conn(0x7f54740ff5e0 msgr2=0x7f5474101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:32.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 -- 172.21.15.39:0/1284787949 shutdown_connections 2024-09-21T01:05:32.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.900+0000 7f547c000640 1 -- 172.21.15.39:0/1284787949 wait complete. 2024-09-21T01:05:32.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.901+0000 7f547c000640 1 Processor -- start 2024-09-21T01:05:32.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.901+0000 7f547c000640 1 -- start start 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.901+0000 7f547c000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474103b70 0x7f547419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f547c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f547419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f547c000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5474109e00 0x7f547419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f547c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5474113ac0 con 0x7f54741094b0 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f547c000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5474113940 con 0x7f5474109e00 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f547c000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5474113c40 con 0x7f5474103b70 2024-09-21T01:05:32.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5478c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f547419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5478c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f547419aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41380/0 (socket says 172.21.15.39:41380) 2024-09-21T01:05:32.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5479600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474103b70 0x7f547419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5478c00640 1 -- 172.21.15.39:0/1897543489 learned_addr learned my addr 172.21.15.39:0/1897543489 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:32.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f547a000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5474109e00 0x7f547419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5479600640 1 -- 172.21.15.39:0/1897543489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5474109e00 msgr2=0x7f547419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5479600640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5474109e00 0x7f547419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5479600640 1 -- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 msgr2=0x7f547419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.902+0000 7f5479600640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f547419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f5479600640 1 -- 172.21.15.39:0/1897543489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741a0420 con 0x7f5474103b70 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f5478c00640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f547419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f5479600640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474103b70 0x7f547419a4b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f546800e9f0 tx=0x7f546800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f5472000640 1 -- 172.21.15.39:0/1897543489 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546800ccc0 con 0x7f5474103b70 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f5472000640 1 -- 172.21.15.39:0/1897543489 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54680108e0 con 0x7f5474103b70 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54741a0710 con 0x7f5474103b70 2024-09-21T01:05:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f5472000640 1 -- 172.21.15.39:0/1897543489 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468018880 con 0x7f5474103b70 2024-09-21T01:05:32.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.903+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54741a6e70 con 0x7f5474103b70 2024-09-21T01:05:32.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.907+0000 7f5482c23740 1 Processor -- start 2024-09-21T01:05:32.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.907+0000 7f5472000640 1 -- 172.21.15.39:0/1897543489 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5468010430 con 0x7f5474103b70 2024-09-21T01:05:32.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.907+0000 7f5482c23740 1 -- start start 2024-09-21T01:05:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5482c23740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 0x55935dc441c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5482c23740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc44700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5482c23740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d87c840 0x55935dc44c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5482c23740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55935dbbb290 con 0x55935d87c840 2024-09-21T01:05:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5482c23740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55935dbaa6c0 con 0x55935d89d520 2024-09-21T01:05:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5482c23740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55935dbae4d0 con 0x55935d874660 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f547a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 0x55935dc441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5479600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc44700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f547a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 0x55935dc441c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35120/0 (socket says 172.21.15.39:35120) 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f547a000640 1 -- 172.21.15.39:0/1773272410 learned_addr learned my addr 172.21.15.39:0/1773272410 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f5478c00640 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d87c840 0x55935dc44c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.908+0000 7f547a000640 1 -- 172.21.15.39:0/1773272410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 msgr2=0x55935dc44700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f547a000640 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc44700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f547a000640 1 -- 172.21.15.39:0/1773272410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d87c840 msgr2=0x55935dc44c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f547a000640 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d87c840 0x55935dc44c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f547a000640 1 -- 172.21.15.39:0/1773272410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55935db37460 con 0x55935d89d520 2024-09-21T01:05:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f547a000640 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 0x55935dc441c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f546c009fd0 tx=0x7f546c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b547b8fc1a6d5b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f5472000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f545c0b6680 0x7f545c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f5466600640 1 -- 172.21.15.39:0/1773272410 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c004d20 con 0x55935d89d520 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f5466600640 1 -- 172.21.15.39:0/1773272410 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f546c004050 con 0x55935d89d520 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.909+0000 7f5466600640 1 -- 172.21.15.39:0/1773272410 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c005280 con 0x55935d89d520 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5472000640 1 -- 172.21.15.39:0/1897543489 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f54680e7e30 con 0x7f5474103b70 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5479600640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f545c0b6680 0x7f545c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 -- 172.21.15.39:0/1773272410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 msgr2=0x55935dc441c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 0x55935dc441c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f546c009fd0 tx=0x7f546c0118d0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 -- 172.21.15.39:0/1773272410 shutdown_connections 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d87c840 0x55935dc44c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc44700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 --2- 172.21.15.39:0/1773272410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d89d520 0x55935dc441c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.910+0000 7f5482c23740 1 -- 172.21.15.39:0/1773272410 >> 172.21.15.39:0/1773272410 conn(0x55935d891910 msgr2=0x55935dc420b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.911+0000 7f5479600640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f545c0b6680 0x7f545c0b8b90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55935dc42e40 tx=0x7f54680eb8e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.911+0000 7f5482c23740 1 -- 172.21.15.39:0/1773272410 shutdown_connections 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.911+0000 7f5482c23740 1 -- 172.21.15.39:0/1773272410 wait complete. 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.912+0000 7f5482c23740 1 -- start start 2024-09-21T01:05:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.912+0000 7f5482c23740 1 Processor -- start 2024-09-21T01:05:32.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc54180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc546c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc5ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55935da0bbb0 con 0x55935d89d520 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55935db456c0 con 0x55935d87c840 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55935dc56c40 con 0x55935d874660 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5482c23740 10 client.0 _ll_register_callbacks cb 0x55935db60490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f547a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc5ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f547a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc5ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41398/0 (socket says 172.21.15.39:41398) 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f547a000640 1 -- 172.21.15.39:0/2928165325 learned_addr learned my addr 172.21.15.39:0/2928165325 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f5478c00640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.913+0000 7f547a000640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc54180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f547a000640 1 -- 172.21.15.39:0/2928165325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 msgr2=0x55935dc54180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f547a000640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc54180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f547a000640 1 -- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 msgr2=0x55935dc546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f547a000640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f547a000640 1 -- 172.21.15.39:0/2928165325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55935d908b00 con 0x55935d89d520 2024-09-21T01:05:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f547a000640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc5ff70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f546c004910 tx=0x7f546c005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6473b27c412b3e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.914+0000 7f545aa00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f545aa00640 1 -- 172.21.15.39:0/2928165325 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c02c020 con 0x55935d89d520 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f546c004050 con 0x55935d89d520 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f545aa00640 1 -- 172.21.15.39:0/2928165325 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f546c004a40 con 0x55935d89d520 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f545aa00640 1 -- 172.21.15.39:0/2928165325 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f546c034050 con 0x55935d89d520 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f5482c23740 10 client.15570 fetch_fsmap learned FSMap version 27 2024-09-21T01:05:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55935d97f960 con 0x55935d89d520 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.915+0000 7f545aa00640 1 -- 172.21.15.39:0/2928165325 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f546c004050 con 0x55935d89d520 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f545aa00640 1 -- 172.21.15.39:0/2928165325 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f546c028450 con 0x55935d89d520 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5482c23740 10 client.15570 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5482c23740 10 client.15570 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5482c23740 20 client.15570 populate_metadata read hostname 'smithi039' 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5482c23740 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d85bea0 0x55935dc65eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5482c23740 4 client.15570 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55935dbbd1e0 con 0x55935d85bea0 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.916+0000 7f5478c00640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d85bea0 0x55935dc65eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.917+0000 7f5478c00640 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d85bea0 0x55935dc65eb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9535e33234d8e8b2 server_cookie=bdd6626394546cd3 in_seq=0 out_seq=0 2024-09-21T01:05:32.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.917+0000 7f545aa00640 10 client.15570 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:32.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f545aa00640 1 -- 172.21.15.39:0/2928165325 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55935dbbd1e0 con 0x55935d85bea0 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f545aa00640 10 client.15570 handle_command_reply: tid=0 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 client.15570 shutdown 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 msgr2=0x55935dc5ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc5ff70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f546c004910 tx=0x7f546c005a90 comp rx=0 tx=0).stop 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 shutdown_connections 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d85bea0 0x55935dc65eb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc5ff70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 --2- 172.21.15.39:0/2928165325 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc54180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f545aa00640 0 client.15570 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 >> 172.21.15.39:0/2928165325 conn(0x55935d891910 msgr2=0x55935dc40650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 shutdown_connections 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.918+0000 7f5482c23740 1 -- 172.21.15.39:0/2928165325 wait complete. 2024-09-21T01:05:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.919+0000 7f5482c23740 20 client.15570 trim_cache size 0 max 16384 2024-09-21T01:05:32.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.937+0000 7f5482c23740 1 Processor -- start 2024-09-21T01:05:32.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.937+0000 7f5482c23740 1 -- start start 2024-09-21T01:05:32.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.937+0000 7f5482c23740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d874660 0x55935dc5fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.937+0000 7f5482c23740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d87c840 0x55935dc603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f5482c23740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc51170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f5482c23740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54741b1650 con 0x55935d89d520 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f5482c23740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f54741aef20 con 0x55935d874660 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f5482c23740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f54741acd70 con 0x55935d87c840 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc51170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d874660 0x55935dc5fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc51170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41406/0 (socket says 172.21.15.39:41406) 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 -- 172.21.15.39:0/1604737287 learned_addr learned my addr 172.21.15.39:0/1604737287 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:32.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f5478c00640 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d87c840 0x55935dc603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 -- 172.21.15.39:0/1604737287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d87c840 msgr2=0x55935dc603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d87c840 0x55935dc603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 -- 172.21.15.39:0/1604737287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d874660 msgr2=0x55935dc5fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d874660 0x55935dc5fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f547a000640 1 -- 172.21.15.39:0/1604737287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55935dc46ad0 con 0x55935d89d520 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.938+0000 7f5478c00640 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d87c840 0x55935dc603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.939+0000 7f547a000640 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc51170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f546c009c30 tx=0x7f546c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611ab916be71afda server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.939+0000 7f5465c00640 1 -- 172.21.15.39:0/1604737287 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c02c070 con 0x55935d89d520 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.939+0000 7f5465c00640 1 -- 172.21.15.39:0/1604737287 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f546c028450 con 0x55935d89d520 2024-09-21T01:05:32.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.939+0000 7f5465c00640 1 -- 172.21.15.39:0/1604737287 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c004050 con 0x55935d89d520 2024-09-21T01:05:32.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 -- 172.21.15.39:0/1604737287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 msgr2=0x55935dc51170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc51170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f546c009c30 tx=0x7f546c0118d0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 -- 172.21.15.39:0/1604737287 shutdown_connections 2024-09-21T01:05:32.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dc51170 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d87c840 0x55935dc603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 --2- 172.21.15.39:0/1604737287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d874660 0x55935dc5fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 -- 172.21.15.39:0/1604737287 >> 172.21.15.39:0/1604737287 conn(0x55935d891910 msgr2=0x55935dc451f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:32.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 -- 172.21.15.39:0/1604737287 shutdown_connections 2024-09-21T01:05:32.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.940+0000 7f5482c23740 1 -- 172.21.15.39:0/1604737287 wait complete. 2024-09-21T01:05:32.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 -- start start 2024-09-21T01:05:32.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 Processor -- start 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc595b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc59af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dcd0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54741ad3e0 con 0x55935d89d520 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f54741ae850 con 0x55935d87c840 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.941+0000 7f5482c23740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f54741adab0 con 0x55935d874660 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5482c23740 10 client.0 _ll_register_callbacks cb 0x7f54680eb7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f547a000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5478c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc59af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f547a000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:54522/0 (socket says 172.21.15.39:54522) 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f547a000640 1 -- 172.21.15.39:0/3205723332 learned_addr learned my addr 172.21.15.39:0/3205723332 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:05:32.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f547a000640 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dcd0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5478c00640 1 -- 172.21.15.39:0/3205723332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 msgr2=0x55935dc595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5478c00640 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5478c00640 1 -- 172.21.15.39:0/3205723332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 msgr2=0x55935dcd0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5478c00640 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dcd0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.942+0000 7f5478c00640 1 -- 172.21.15.39:0/3205723332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55935dcd0ca0 con 0x55935d87c840 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.943+0000 7f5478c00640 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc59af0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5460031440 tx=0x7f54600043d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc3f028348ba88f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:05:32.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.943+0000 7f545aa00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.943+0000 7f545aa00640 1 -- 172.21.15.39:0/3205723332 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546004b030 con 0x55935d87c840 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.943+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f546c028450 con 0x55935d87c840 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.943+0000 7f545aa00640 1 -- 172.21.15.39:0/3205723332 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5460005020 con 0x55935d87c840 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.943+0000 7f545aa00640 1 -- 172.21.15.39:0/3205723332 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5460035040 con 0x55935d87c840 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 10 client.25607 fetch_fsmap learned FSMap version 27 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55935dcd1fc0 con 0x55935d87c840 2024-09-21T01:05:32.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f545aa00640 1 -- 172.21.15.39:0/3205723332 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f546c028450 con 0x55935d87c840 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f545aa00640 1 -- 172.21.15.39:0/3205723332 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5460004770 con 0x55935d87c840 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 10 client.25607 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 10 client.25607 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 20 client.25607 populate_metadata read hostname 'smithi039' 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d831250 0x55935dcd47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 4 client.25607 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ce59dea0", "release": true, "await": true}] 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.944+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ce59dea0", "release": true, "await": true}) -- 0x55935dbb8c90 con 0x55935d831250 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.945+0000 7f547a000640 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d831250 0x55935dcd47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.945+0000 7f547a000640 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d831250 0x55935dcd47b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cab797d28bcc640 server_cookie=9ca07ccce6436676 in_seq=0 out_seq=0 2024-09-21T01:05:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.945+0000 7f545aa00640 10 client.25607 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:32.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.948+0000 7f545aa00640 1 -- 172.21.15.39:0/3205723332 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55935dbb8c90 con 0x55935d831250 2024-09-21T01:05:32.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.948+0000 7f545aa00640 10 client.25607 handle_command_reply: tid=0 2024-09-21T01:05:32.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.948+0000 7f5482c23740 1 client.25607 shutdown 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.948+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 msgr2=0x55935dc59af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.948+0000 7f5482c23740 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc59af0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5460031440 tx=0x7f54600043d0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 shutdown_connections 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55935d831250 0x55935dcd47b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55935d89d520 0x55935dcd0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935d87c840 0x55935dc59af0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 --2- 172.21.15.39:0/3205723332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55935d874660 0x55935dc595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 >> 172.21.15.39:0/3205723332 conn(0x55935d891910 msgr2=0x55935dc451f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f545aa00640 0 client.25607 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 shutdown_connections 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 1 -- 172.21.15.39:0/3205723332 wait complete. 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.949+0000 7f5482c23740 20 client.25607 trim_cache size 0 max 16384 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 27, 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:05:32.951 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 8, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "ce59dea0": { 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 8, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 137.0, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:05:32.952 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:05:32.953 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f545c0b6680 msgr2=0x7f545c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f545c0b6680 0x7f545c0b8b90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55935dc42e40 tx=0x7f54680eb8e0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474103b70 msgr2=0x7f547419a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474103b70 0x7f547419a4b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f546800e9f0 tx=0x7f546800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 shutdown_connections 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f545c0b6680 0x7f545c0b8b90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5474109e00 0x7f547419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54741094b0 0x7f547419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 --2- 172.21.15.39:0/1897543489 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5474103b70 0x7f547419a4b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.950+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 >> 172.21.15.39:0/1897543489 conn(0x7f54740ff5e0 msgr2=0x7f5474101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.951+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 shutdown_connections 2024-09-21T01:05:32.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:05:32.951+0000 7f547c000640 1 -- 172.21.15.39:0/1897543489 wait complete. 2024-09-21T01:05:32.963 INFO:tasks.workunit.client.0.smithi039.stderr:+ dbench 1 2024-09-21T01:05:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T01:05:32.978 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-21T01:05:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T01:05:32.999 INFO:tasks.workunit.client.0.smithi039.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-21T01:05:33.100 INFO:tasks.workunit.client.0.smithi039.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-21T01:05:33.100 INFO:tasks.workunit.client.0.smithi039.stdout:releasing clients 2024-09-21T01:05:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:33 smithi039 ceph-mon[30415]: pgmap v450: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:05:34.005 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ce59dea0', total seconds elapsed: 21.5 2024-09-21T01:05:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:33 smithi150 ceph-mon[34945]: pgmap v450: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:05:34.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 3177 34.12 MB/sec warmup 1 sec latency 8.688 ms 2024-09-21T01:05:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:33 smithi160 ceph-mon[34696]: pgmap v450: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:05:35.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 6037 23.92 MB/sec warmup 2 sec latency 9.614 ms 2024-09-21T01:05:36.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:35 smithi150 ceph-mon[34945]: pgmap v451: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:05:36.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 8824 21.49 MB/sec warmup 3 sec latency 11.420 ms 2024-09-21T01:05:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:35 smithi160 ceph-mon[34696]: pgmap v451: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:05:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:35 smithi039 ceph-mon[30415]: pgmap v451: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-21T01:05:37.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 11438 19.68 MB/sec warmup 4 sec latency 18.570 ms 2024-09-21T01:05:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:37 smithi150 ceph-mon[34945]: pgmap v452: 129 pgs: 129 active+clean; 3.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 160 KiB/s wr, 29 op/s 2024-09-21T01:05:38.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 14645 19.48 MB/sec warmup 5 sec latency 6.386 ms 2024-09-21T01:05:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:37 smithi160 ceph-mon[34696]: pgmap v452: 129 pgs: 129 active+clean; 3.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 160 KiB/s wr, 29 op/s 2024-09-21T01:05:38.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:37 smithi039 ceph-mon[30415]: pgmap v452: 129 pgs: 129 active+clean; 3.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 160 KiB/s wr, 29 op/s 2024-09-21T01:05:39.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 17950 19.37 MB/sec warmup 6 sec latency 11.398 ms 2024-09-21T01:05:40.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:39 smithi150 ceph-mon[34945]: pgmap v453: 129 pgs: 129 active+clean; 4.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 348 KiB/s wr, 39 op/s 2024-09-21T01:05:40.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 20269 17.77 MB/sec warmup 7 sec latency 21.371 ms 2024-09-21T01:05:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:39 smithi160 ceph-mon[34696]: pgmap v453: 129 pgs: 129 active+clean; 4.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 348 KiB/s wr, 39 op/s 2024-09-21T01:05:40.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:39 smithi039 ceph-mon[30415]: pgmap v453: 129 pgs: 129 active+clean; 4.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 348 KiB/s wr, 39 op/s 2024-09-21T01:05:41.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 22918 17.62 MB/sec warmup 8 sec latency 11.680 ms 2024-09-21T01:05:42.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:41 smithi150 ceph-mon[34945]: pgmap v454: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 646 KiB/s wr, 84 op/s 2024-09-21T01:05:42.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 25682 17.26 MB/sec warmup 9 sec latency 11.815 ms 2024-09-21T01:05:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:41 smithi160 ceph-mon[34696]: pgmap v454: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 646 KiB/s wr, 84 op/s 2024-09-21T01:05:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:41 smithi039 ceph-mon[30415]: pgmap v454: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 646 KiB/s wr, 84 op/s 2024-09-21T01:05:43.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 28834 17.40 MB/sec warmup 10 sec latency 14.317 ms 2024-09-21T01:05:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:43 smithi039 ceph-mon[30415]: pgmap v455: 129 pgs: 129 active+clean; 12 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-09-21T01:05:44.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:43 smithi150 ceph-mon[34945]: pgmap v455: 129 pgs: 129 active+clean; 12 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-09-21T01:05:44.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 32035 17.42 MB/sec warmup 11 sec latency 12.265 ms 2024-09-21T01:05:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:43 smithi160 ceph-mon[34696]: pgmap v455: 129 pgs: 129 active+clean; 12 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-09-21T01:05:45.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 34546 16.76 MB/sec warmup 12 sec latency 16.174 ms 2024-09-21T01:05:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:45 smithi150 ceph-mon[34945]: pgmap v456: 129 pgs: 129 active+clean; 13 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-09-21T01:05:46.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 36918 16.71 MB/sec warmup 13 sec latency 20.100 ms 2024-09-21T01:05:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:45 smithi160 ceph-mon[34696]: pgmap v456: 129 pgs: 129 active+clean; 13 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-09-21T01:05:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:45 smithi039 ceph-mon[30415]: pgmap v456: 129 pgs: 129 active+clean; 13 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-09-21T01:05:47.100 INFO:tasks.workunit.client.0.smithi039.stdout: 1 39778 16.54 MB/sec warmup 14 sec latency 12.316 ms 2024-09-21T01:05:48.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:47 smithi150 ceph-mon[34945]: pgmap v457: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-09-21T01:05:48.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 42962 16.70 MB/sec warmup 15 sec latency 9.374 ms 2024-09-21T01:05:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:47 smithi160 ceph-mon[34696]: pgmap v457: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-09-21T01:05:48.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:47 smithi039 ceph-mon[30415]: pgmap v457: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-09-21T01:05:49.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 45770 16.48 MB/sec warmup 16 sec latency 20.837 ms 2024-09-21T01:05:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:49 smithi150 ceph-mon[34945]: pgmap v458: 129 pgs: 129 active+clean; 19 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-09-21T01:05:50.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 48125 16.18 MB/sec warmup 17 sec latency 12.139 ms 2024-09-21T01:05:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:49 smithi160 ceph-mon[34696]: pgmap v458: 129 pgs: 129 active+clean; 19 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-09-21T01:05:50.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:49 smithi039 ceph-mon[30415]: pgmap v458: 129 pgs: 129 active+clean; 19 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-09-21T01:05:51.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 50494 16.07 MB/sec warmup 18 sec latency 12.606 ms 2024-09-21T01:05:52.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:51 smithi150 ceph-mon[34945]: pgmap v459: 129 pgs: 129 active+clean; 19 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-09-21T01:05:52.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 53498 16.18 MB/sec warmup 19 sec latency 14.919 ms 2024-09-21T01:05:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:51 smithi160 ceph-mon[34696]: pgmap v459: 129 pgs: 129 active+clean; 19 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-09-21T01:05:52.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:51 smithi039 ceph-mon[30415]: pgmap v459: 129 pgs: 129 active+clean; 19 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-09-21T01:05:53.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:52 smithi150 ceph-mon[34945]: pgmap v460: 129 pgs: 129 active+clean; 22 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.5 MiB/s wr, 173 op/s 2024-09-21T01:05:53.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 56111 15.95 MB/sec warmup 20 sec latency 11.943 ms 2024-09-21T01:05:53.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:52 smithi160 ceph-mon[34696]: pgmap v460: 129 pgs: 129 active+clean; 22 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.5 MiB/s wr, 173 op/s 2024-09-21T01:05:53.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:52 smithi039 ceph-mon[30415]: pgmap v460: 129 pgs: 129 active+clean; 22 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.5 MiB/s wr, 173 op/s 2024-09-21T01:05:54.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 58389 15.82 MB/sec warmup 21 sec latency 12.890 ms 2024-09-21T01:05:55.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 60653 15.71 MB/sec warmup 22 sec latency 112.435 ms 2024-09-21T01:05:55.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:55 smithi150 ceph-mon[34945]: pgmap v461: 129 pgs: 129 active+clean; 27 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-09-21T01:05:55.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:55 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:05:55.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:55 smithi160 ceph-mon[34696]: pgmap v461: 129 pgs: 129 active+clean; 27 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-09-21T01:05:55.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:55 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:05:55.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:55 smithi039 ceph-mon[30415]: pgmap v461: 129 pgs: 129 active+clean; 27 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-09-21T01:05:55.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:55 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:05:56.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 62784 15.36 MB/sec warmup 23 sec latency 21.897 ms 2024-09-21T01:05:56.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:56 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:05:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:05:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:05:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:05:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:56 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:05:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:05:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:05:56.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:05:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:56 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:05:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:05:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:05:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:05:57.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 64741 15.24 MB/sec warmup 24 sec latency 14.224 ms 2024-09-21T01:05:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:57 smithi150 ceph-mon[34945]: pgmap v462: 129 pgs: 129 active+clean; 29 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-09-21T01:05:57.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:57 smithi160 ceph-mon[34696]: pgmap v462: 129 pgs: 129 active+clean; 29 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-09-21T01:05:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:57 smithi039 ceph-mon[30415]: pgmap v462: 129 pgs: 129 active+clean; 29 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-09-21T01:05:58.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 67653 15.36 MB/sec warmup 25 sec latency 10.270 ms 2024-09-21T01:05:59.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 69820 15.05 MB/sec warmup 26 sec latency 16.348 ms 2024-09-21T01:05:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:05:59 smithi150 ceph-mon[34945]: pgmap v463: 129 pgs: 129 active+clean; 30 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.5 MiB/s wr, 188 op/s 2024-09-21T01:05:59.637 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:05:59 smithi160 ceph-mon[34696]: pgmap v463: 129 pgs: 129 active+clean; 30 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.5 MiB/s wr, 188 op/s 2024-09-21T01:05:59.657 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:05:59 smithi039 ceph-mon[30415]: pgmap v463: 129 pgs: 129 active+clean; 30 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.5 MiB/s wr, 188 op/s 2024-09-21T01:06:00.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 71768 14.96 MB/sec warmup 27 sec latency 34.024 ms 2024-09-21T01:06:01.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 73271 14.63 MB/sec warmup 28 sec latency 258.014 ms 2024-09-21T01:06:01.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:01 smithi150 ceph-mon[34945]: pgmap v464: 129 pgs: 129 active+clean; 33 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-09-21T01:06:01.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:01 smithi160 ceph-mon[34696]: pgmap v464: 129 pgs: 129 active+clean; 33 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-09-21T01:06:01.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:01 smithi039 ceph-mon[30415]: pgmap v464: 129 pgs: 129 active+clean; 33 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-09-21T01:06:02.102 INFO:tasks.workunit.client.0.smithi039.stdout: 1 74686 14.53 MB/sec warmup 29 sec latency 59.536 ms 2024-09-21T01:06:03.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 76343 14.27 MB/sec warmup 30 sec latency 16.657 ms 2024-09-21T01:06:03.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:03 smithi150 ceph-mon[34945]: pgmap v465: 129 pgs: 129 active+clean; 35 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 240 op/s 2024-09-21T01:06:03.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:03 smithi160 ceph-mon[34696]: pgmap v465: 129 pgs: 129 active+clean; 35 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 240 op/s 2024-09-21T01:06:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:03 smithi039 ceph-mon[30415]: pgmap v465: 129 pgs: 129 active+clean; 35 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 240 op/s 2024-09-21T01:06:04.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 78644 14.25 MB/sec warmup 31 sec latency 10.600 ms 2024-09-21T01:06:05.102 INFO:tasks.workunit.client.0.smithi039.stdout: 1 79145 13.86 MB/sec warmup 32 sec latency 475.184 ms 2024-09-21T01:06:05.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:05 smithi150 ceph-mon[34945]: pgmap v466: 129 pgs: 129 active+clean; 37 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.6 MiB/s wr, 243 op/s 2024-09-21T01:06:05.577 INFO:tasks.quiescer.fs.[cephfs]:Total duration (55.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:06:05.577 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.8 2024-09-21T01:06:05.577 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:06:05.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:05 smithi160 ceph-mon[34696]: pgmap v466: 129 pgs: 129 active+clean; 37 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.6 MiB/s wr, 243 op/s 2024-09-21T01:06:05.631 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:05 smithi039 ceph-mon[30415]: pgmap v466: 129 pgs: 129 active+clean; 37 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.6 MiB/s wr, 243 op/s 2024-09-21T01:06:06.001 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:06:06.102 INFO:tasks.workunit.client.0.smithi039.stdout: 1 81763 13.92 MB/sec warmup 33 sec latency 13.670 ms 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.666+0000 7fd8aea00640 1 -- 172.21.15.39:0/3109471680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108480 msgr2=0x7fd8a8108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.666+0000 7fd8aea00640 1 --2- 172.21.15.39:0/3109471680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108480 0x7fd8a8108890 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd894009900 tx=0x7fd89402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 -- 172.21.15.39:0/3109471680 shutdown_connections 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 --2- 172.21.15.39:0/3109471680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8108dd0 0x7fd8a810f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 --2- 172.21.15.39:0/3109471680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108480 0x7fd8a8108890 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 --2- 172.21.15.39:0/3109471680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8102b40 0x7fd8a8102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 -- 172.21.15.39:0/3109471680 >> 172.21.15.39:0/3109471680 conn(0x7fd8a8079750 msgr2=0x7fd8a8079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 -- 172.21.15.39:0/3109471680 shutdown_connections 2024-09-21T01:06:06.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.667+0000 7fd8aea00640 1 -- 172.21.15.39:0/3109471680 wait complete. 2024-09-21T01:06:06.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.668+0000 7fd8aea00640 1 Processor -- start 2024-09-21T01:06:06.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.668+0000 7fd8aea00640 1 -- start start 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.668+0000 7fd8aea00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 0x7fd8a80ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8aea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 0x7fd8a80ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8aea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108dd0 0x7fd8a8100120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8aea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81129f0 con 0x7fd8a8108480 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8aea00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8112870 con 0x7fd8a8102b40 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8aea00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8112b70 con 0x7fd8a8108dd0 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 0x7fd8a80ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 0x7fd8a80ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 0x7fd8a80ff6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41386/0 (socket says 172.21.15.39:41386) 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 0x7fd8a80ffbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55876/0 (socket says 172.21.15.39:55876) 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 -- 172.21.15.39:0/619874218 learned_addr learned my addr 172.21.15.39:0/619874218 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8aca00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108dd0 0x7fd8a8100120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 -- 172.21.15.39:0/619874218 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108dd0 msgr2=0x7fd8a8100120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:06.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108dd0 0x7fd8a8100120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 -- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 msgr2=0x7fd8a80ffbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:06.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 0x7fd8a80ffbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7e00640 1 -- 172.21.15.39:0/619874218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a81a86f0 con 0x7fd8a8102b40 2024-09-21T01:06:06.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.669+0000 7fd8a7400640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 0x7fd8a80ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:06:06.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.670+0000 7fd8a7e00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 0x7fd8a80ff6a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd89800ea60 tx=0x7fd89800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:06.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.670+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89800cde0 con 0x7fd8a8102b40 2024-09-21T01:06:06.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.670+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8980035f0 con 0x7fd8a8102b40 2024-09-21T01:06:06.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.670+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898010690 con 0x7fd8a8102b40 2024-09-21T01:06:06.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.670+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a81a89e0 con 0x7fd8a8102b40 2024-09-21T01:06:06.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.670+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a819a310 con 0x7fd8a8102b40 2024-09-21T01:06:06.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.672+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a8103dc0 con 0x7fd8a8102b40 2024-09-21T01:06:06.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.674+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fd8980037a0 con 0x7fd8a8102b40 2024-09-21T01:06:06.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.676+0000 7fd8a4c00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd88c0b6680 0x7fd88c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:06.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.676+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd8980e7de0 con 0x7fd8a8102b40 2024-09-21T01:06:06.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.676+0000 7fd8a7400640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd88c0b6680 0x7fd88c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:06.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.677+0000 7fd8a7400640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd88c0b6680 0x7fd88c0b8b90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd894031440 tx=0x7fd894005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:06.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.680+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd8980adf50 con 0x7fd8a8102b40 2024-09-21T01:06:06.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.949+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8a810bbd0 con 0x7fd8a8102b40 2024-09-21T01:06:06.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.950+0000 7fd8a4c00640 1 -- 172.21.15.39:0/619874218 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11914 (secure 0 0 0) 0x7fd8980b2330 con 0x7fd8a8102b40 2024-09-21T01:06:06.952 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:06:06.952 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"btime":"2024-09-21T01:05:55:033931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:05:54.497271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:06:06.952 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.952+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd88c0b6680 msgr2=0x7fd88c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd88c0b6680 0x7fd88c0b8b90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd894031440 tx=0x7fd894005a20 comp rx=0 tx=0).stop 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 msgr2=0x7fd8a80ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 0x7fd8a80ff6a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd89800ea60 tx=0x7fd89800ef20 comp rx=0 tx=0).stop 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 shutdown_connections 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8a8108dd0 0x7fd8a8100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a8108480 0x7fd8a80ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd88c0b6680 0x7fd88c0b8b90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 --2- 172.21.15.39:0/619874218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8a8102b40 0x7fd8a80ff6a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:06.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 >> 172.21.15.39:0/619874218 conn(0x7fd8a8079750 msgr2=0x7fd8a8101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:06.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 shutdown_connections 2024-09-21T01:06:06.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:06.953+0000 7fd8aea00640 1 -- 172.21.15.39:0/619874218 wait complete. 2024-09-21T01:06:07.102 INFO:tasks.workunit.client.0.smithi039.stdout: 1 83224 13.71 MB/sec warmup 34 sec latency 17.865 ms 2024-09-21T01:06:07.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:07 smithi150 ceph-mon[34945]: pgmap v467: 129 pgs: 129 active+clean; 43 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.7 MiB/s wr, 197 op/s 2024-09-21T01:06:07.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:07 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/619874218' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:07.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:07 smithi160 ceph-mon[34696]: pgmap v467: 129 pgs: 129 active+clean; 43 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.7 MiB/s wr, 197 op/s 2024-09-21T01:06:07.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:07 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/619874218' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:07 smithi039 ceph-mon[30415]: pgmap v467: 129 pgs: 129 active+clean; 43 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.7 MiB/s wr, 197 op/s 2024-09-21T01:06:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:07 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/619874218' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:08.065 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0' 2024-09-21T01:06:08.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0 2024-09-21T01:06:08.101 INFO:tasks.workunit.client.0.smithi039.stdout: 1 85492 13.70 MB/sec warmup 35 sec latency 37.163 ms 2024-09-21T01:06:08.472 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:06:09.102 INFO:tasks.workunit.client.0.smithi039.stdout: 1 87574 13.53 MB/sec warmup 36 sec latency 14.618 ms 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 -- 172.21.15.39:0/2171773541 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 msgr2=0x7f728c112680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 --2- 172.21.15.39:0/2171773541 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 0x7f728c112680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7284009ca0 tx=0x7f728402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 -- 172.21.15.39:0/2171773541 shutdown_connections 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 --2- 172.21.15.39:0/2171773541 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 0x7f728c112680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 --2- 172.21.15.39:0/2171773541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c10b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 --2- 172.21.15.39:0/2171773541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f728c105bc0 0x7f728c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 -- 172.21.15.39:0/2171773541 >> 172.21.15.39:0/2171773541 conn(0x7f728c101610 msgr2=0x7f728c103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 -- 172.21.15.39:0/2171773541 shutdown_connections 2024-09-21T01:06:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.250+0000 7f728be00640 1 -- 172.21.15.39:0/2171773541 wait complete. 2024-09-21T01:06:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.251+0000 7f728be00640 1 Processor -- start 2024-09-21T01:06:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.251+0000 7f728be00640 1 -- start start 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f728c105bc0 0x7f728c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 0x7f728c1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f728c115b10 con 0x7f728c105bc0 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f728c115990 con 0x7f728c10b500 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728be00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f728c115c90 con 0x7f728c10be50 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41412/0 (socket says 172.21.15.39:41412) 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 0x7f728c1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728a000640 1 -- 172.21.15.39:0/3968277862 learned_addr learned my addr 172.21.15.39:0/3968277862 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.252+0000 7f728aa00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f728c105bc0 0x7f728c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728a000640 1 -- 172.21.15.39:0/3968277862 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 msgr2=0x7f728c1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728a000640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 0x7f728c1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728a000640 1 -- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f728c105bc0 msgr2=0x7f728c19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728a000640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f728c105bc0 0x7f728c19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728a000640 1 -- 172.21.15.39:0/3968277862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c1a23f0 con 0x7f728c10b500 2024-09-21T01:06:09.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728a000640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c19ca20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f727800e9f0 tx=0x7f727800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:09.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f7283400640 1 -- 172.21.15.39:0/3968277862 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727800ccc0 con 0x7f728c10b500 2024-09-21T01:06:09.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f7283400640 1 -- 172.21.15.39:0/3968277862 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72780108e0 con 0x7f728c10b500 2024-09-21T01:06:09.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f7283400640 1 -- 172.21.15.39:0/3968277862 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72780187c0 con 0x7f728c10b500 2024-09-21T01:06:09.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c1a26e0 con 0x7f728c10b500 2024-09-21T01:06:09.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.253+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c1a8e40 con 0x7f728c10b500 2024-09-21T01:06:09.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 Processor -- start 2024-09-21T01:06:09.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 -- start start 2024-09-21T01:06:09.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7283400640 1 -- 172.21.15.39:0/3968277862 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7278010430 con 0x7f728c10b500 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dbc8840 0x55cb6df70510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df70a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df70f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cb6dee5ca0 con 0x55cb6dbc8840 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55cb6ded50d0 con 0x55cb6db9f980 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f7298841740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55cb6ded8ee0 con 0x55cb6dba7b60 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.258+0000 7f728a000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df70f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dbc8840 0x55cb6df70510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df70a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df70a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41416/0 (socket says 172.21.15.39:41416) 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dbc8840 0x55cb6df70510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55898/0 (socket says 172.21.15.39:55898) 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 -- 172.21.15.39:0/1721449227 learned_addr learned my addr 172.21.15.39:0/1721449227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:09.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728a000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df70f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59220/0 (socket says 172.21.15.39:59220) 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 -- 172.21.15.39:0/1721449227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 msgr2=0x55cb6df70f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df70f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 -- 172.21.15.39:0/1721449227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dbc8840 msgr2=0x55cb6df70510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dbc8840 0x55cb6df70510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.259+0000 7f728aa00640 1 -- 172.21.15.39:0/1721449227 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb6de87400 con 0x55cb6db9f980 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.260+0000 7f728aa00640 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df70a50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f727c009cf0 tx=0x7f727c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66eb4312a3558ca9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.260+0000 7f7277e00640 1 -- 172.21.15.39:0/1721449227 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c004230 con 0x55cb6db9f980 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.260+0000 7f7277e00640 1 -- 172.21.15.39:0/1721449227 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f727c0043d0 con 0x55cb6db9f980 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.260+0000 7f7277e00640 1 -- 172.21.15.39:0/1721449227 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c019880 con 0x55cb6db9f980 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.260+0000 7f7283400640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72700b66f0 0x7f72700b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7283400640 1 -- 172.21.15.39:0/3968277862 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f72780e7d20 con 0x7f728c10b500 2024-09-21T01:06:09.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f728b400640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72700b66f0 0x7f72700b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 -- 172.21.15.39:0/1721449227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 msgr2=0x55cb6df70a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df70a50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f727c009cf0 tx=0x7f727c011810 comp rx=0 tx=0).stop 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 -- 172.21.15.39:0/1721449227 shutdown_connections 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df70f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df70a50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 --2- 172.21.15.39:0/1721449227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dbc8840 0x55cb6df70510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 -- 172.21.15.39:0/1721449227 >> 172.21.15.39:0/1721449227 conn(0x55cb6dbbc990 msgr2=0x55cb6df6e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f7298841740 1 -- 172.21.15.39:0/1721449227 shutdown_connections 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.261+0000 7f728b400640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72700b66f0 0x7f72700b8c00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f728402f500 tx=0x7f728402f310 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.262+0000 7f7298841740 1 -- 172.21.15.39:0/1721449227 wait complete. 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.262+0000 7f7298841740 1 -- start start 2024-09-21T01:06:09.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.262+0000 7f7298841740 1 Processor -- start 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df7bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dba7b60 0x55cb6df7c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df88040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cb6dd367a0 con 0x55cb6dba7b60 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55cb6de7aa10 con 0x55cb6db9f980 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55cb6df7ecf0 con 0x55cb6dbc8840 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f7298841740 10 client.0 _ll_register_callbacks cb 0x55cb6de8abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:06:09.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f728aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df7bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f728aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df7bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41422/0 (socket says 172.21.15.39:41422) 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f728aa00640 1 -- 172.21.15.39:0/1069591487 learned_addr learned my addr 172.21.15.39:0/1069591487 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f728b400640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df88040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.263+0000 7f728a000640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dba7b60 0x55cb6df7c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728aa00640 1 -- 172.21.15.39:0/1069591487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 msgr2=0x55cb6df88040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728aa00640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df88040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728aa00640 1 -- 172.21.15.39:0/1069591487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dba7b60 msgr2=0x55cb6df7c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728aa00640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dba7b60 0x55cb6df7c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728aa00640 1 -- 172.21.15.39:0/1069591487 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb6dc42b90 con 0x55cb6db9f980 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728a000640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dba7b60 0x55cb6df7c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f728aa00640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df7bc40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f727c019460 tx=0x7f727c003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73f3440200206dc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:09.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f7274c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f7274c00640 1 -- 172.21.15.39:0/1069591487 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c027070 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f7274c00640 1 -- 172.21.15.39:0/1069591487 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f727c005240 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.264+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f727c0043d0 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7274c00640 1 -- 172.21.15.39:0/1069591487 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f727c033050 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 10 client.25625 fetch_fsmap learned FSMap version 28 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cb6df8aa00 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7274c00640 1 -- 172.21.15.39:0/1069591487 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f727c0043d0 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7274c00640 1 -- 172.21.15.39:0/1069591487 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f727c004b70 con 0x55cb6db9f980 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 10 client.25625 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 10 client.25625 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 20 client.25625 populate_metadata read hostname 'smithi039' 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db871c0 0x55cb6df8ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 4 client.25625 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:06:09.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.265+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb6dee7bf0 con 0x55cb6db871c0 2024-09-21T01:06:09.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.266+0000 7f728a000640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db871c0 0x55cb6df8ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.266+0000 7f728a000640 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db871c0 0x55cb6df8ddc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7413e3bbbb82f0b server_cookie=374a814c1bb3fffb in_seq=0 out_seq=0 2024-09-21T01:06:09.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.266+0000 7f7274c00640 10 client.25625 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:09.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7274c00640 1 -- 172.21.15.39:0/1069591487 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cb6dee7bf0 con 0x55cb6db871c0 2024-09-21T01:06:09.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7274c00640 10 client.25625 handle_command_reply: tid=0 2024-09-21T01:06:09.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 client.25625 shutdown 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 msgr2=0x55cb6df7bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df7bc40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f727c019460 tx=0x7f727c003820 comp rx=0 tx=0).stop 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 shutdown_connections 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db871c0 0x55cb6df8ddc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df88040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6dba7b60 0x55cb6df7c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 --2- 172.21.15.39:0/1069591487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6db9f980 0x55cb6df7bc40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7274c00640 0 client.25625 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.267+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 >> 172.21.15.39:0/1069591487 conn(0x55cb6dbbc990 msgr2=0x55cb6df736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.268+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 shutdown_connections 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.268+0000 7f7298841740 1 -- 172.21.15.39:0/1069591487 wait complete. 2024-09-21T01:06:09.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.268+0000 7f7298841740 20 client.25625 trim_cache size 0 max 16384 2024-09-21T01:06:09.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.288+0000 7f7298841740 1 Processor -- start 2024-09-21T01:06:09.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.288+0000 7f7298841740 1 -- start start 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.288+0000 7f7298841740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df7db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.288+0000 7f7298841740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dba7b60 0x55cb6df87f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.288+0000 7f7298841740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df7b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f7298841740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1b5720 con 0x55cb6db9f980 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f7298841740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1b2ff0 con 0x55cb6dba7b60 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f7298841740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1b0e40 con 0x55cb6dbc8840 2024-09-21T01:06:09.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df7db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df7b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df7db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55912/0 (socket says 172.21.15.39:55912) 2024-09-21T01:06:09.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df7b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59230/0 (socket says 172.21.15.39:59230) 2024-09-21T01:06:09.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 -- 172.21.15.39:0/3522923409 learned_addr learned my addr 172.21.15.39:0/3522923409 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:09.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728a000640 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dba7b60 0x55cb6df87f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 -- 172.21.15.39:0/3522923409 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 msgr2=0x55cb6df7b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df7b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 -- 172.21.15.39:0/3522923409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dba7b60 msgr2=0x55cb6df87f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dba7b60 0x55cb6df87f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728aa00640 1 -- 172.21.15.39:0/3522923409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb6df7cf00 con 0x55cb6db9f980 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.289+0000 7f728b400640 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df7b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.290+0000 7f728aa00640 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df7db20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f727c019490 tx=0x7f727c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fe4427c4623cda server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.290+0000 7f7277400640 1 -- 172.21.15.39:0/3522923409 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c03c070 con 0x55cb6db9f980 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.290+0000 7f7277400640 1 -- 172.21.15.39:0/3522923409 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f727c009910 con 0x55cb6db9f980 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.290+0000 7f7277400640 1 -- 172.21.15.39:0/3522923409 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c027040 con 0x55cb6db9f980 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 -- 172.21.15.39:0/3522923409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 msgr2=0x55cb6df7db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df7db20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f727c019490 tx=0x7f727c019710 comp rx=0 tx=0).stop 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 -- 172.21.15.39:0/3522923409 shutdown_connections 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dbc8840 0x55cb6df7b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dba7b60 0x55cb6df87f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 --2- 172.21.15.39:0/3522923409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df7db20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 -- 172.21.15.39:0/3522923409 >> 172.21.15.39:0/3522923409 conn(0x55cb6dbbc990 msgr2=0x55cb6df7b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 -- 172.21.15.39:0/3522923409 shutdown_connections 2024-09-21T01:06:09.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.291+0000 7f7298841740 1 -- 172.21.15.39:0/3522923409 wait complete. 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 -- start start 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 Processor -- start 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df80230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df80770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 0x55cb6dff77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1b14b0 con 0x55cb6db9f980 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1b2920 con 0x55cb6dbc8840 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1b1b80 con 0x55cb6dba7b60 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.292+0000 7f7298841740 10 client.0 _ll_register_callbacks cb 0x7f7278005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 0x55cb6dff77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df80230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 0x55cb6dff77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41434/0 (socket says 172.21.15.39:41434) 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df80230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55918/0 (socket says 172.21.15.39:55918) 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 -- 172.21.15.39:0/556313918 learned_addr learned my addr 172.21.15.39:0/556313918 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728a000640 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df80770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 -- 172.21.15.39:0/556313918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 msgr2=0x55cb6df80770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df80770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 -- 172.21.15.39:0/556313918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 msgr2=0x55cb6df80230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df80230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 -- 172.21.15.39:0/556313918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb6df80f60 con 0x55cb6dbc8840 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f728b400640 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 0x55cb6dff77c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7284009ca0 tx=0x7f72840095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e0256cb9f309a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.293+0000 7f7274c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:06:09.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7274c00640 1 -- 172.21.15.39:0/556313918 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284059060 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f727c009910 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7274c00640 1 -- 172.21.15.39:0/556313918 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7284004040 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7274c00640 1 -- 172.21.15.39:0/556313918 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7284060050 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7298841740 10 client.25631 fetch_fsmap learned FSMap version 28 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cb6dff8d10 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7274c00640 1 -- 172.21.15.39:0/556313918 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f727c009910 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.294+0000 7f7274c00640 1 -- 172.21.15.39:0/556313918 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7284053070 con 0x55cb6dbc8840 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7298841740 10 client.25631 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7298841740 10 client.25631 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7298841740 20 client.25631 populate_metadata read hostname 'smithi039' 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7298841740 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db5c570 0x55cb6dffb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7298841740 4 client.25631 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}] 2024-09-21T01:06:09.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}) -- 0x55cb6dee36a0 con 0x55cb6db5c570 2024-09-21T01:06:09.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f728aa00640 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db5c570 0x55cb6dffb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:09.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f728aa00640 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db5c570 0x55cb6dffb5b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a90075061c94d server_cookie=a4d98e687ac559fe in_seq=0 out_seq=0 2024-09-21T01:06:09.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.295+0000 7f7274c00640 10 client.25631 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:09.482 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:09 smithi039 ceph-mon[30415]: pgmap v468: 129 pgs: 129 active+clean; 58 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 3.0 MiB/s wr, 241 op/s 2024-09-21T01:06:09.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:09 smithi160 ceph-mon[34696]: pgmap v468: 129 pgs: 129 active+clean; 58 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 3.0 MiB/s wr, 241 op/s 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7274c00640 1 -- 172.21.15.39:0/556313918 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cb6dee36a0 con 0x55cb6db5c570 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7274c00640 10 client.25631 handle_command_reply: tid=0 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 client.25631 shutdown 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 msgr2=0x55cb6dff77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 0x55cb6dff77c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7284009ca0 tx=0x7f72840095c0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 shutdown_connections 2024-09-21T01:06:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55cb6db5c570 0x55cb6dffb5b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cb6dbc8840 0x55cb6dff77c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55cb6dba7b60 0x55cb6df80770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7274c00640 0 client.25631 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 --2- 172.21.15.39:0/556313918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb6db9f980 0x55cb6df80230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 >> 172.21.15.39:0/556313918 conn(0x55cb6dbbc990 msgr2=0x55cb6df7d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 shutdown_connections 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 1 -- 172.21.15.39:0/556313918 wait complete. 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f7298841740 20 client.25631 trim_cache size 0 max 16384 2024-09-21T01:06:09.632 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 28, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 10, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "a7e3cae4": { 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 10, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 147.8, 2024-09-21T01:06:09.633 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:09.634 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72700b66f0 msgr2=0x7f72700b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72700b66f0 0x7f72700b8c00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f728402f500 tx=0x7f728402f310 comp rx=0 tx=0).stop 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 msgr2=0x7f728c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c19ca20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f727800e9f0 tx=0x7f727800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 shutdown_connections 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f728c10be50 0x7f728c1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f72700b66f0 0x7f72700b8c00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f728c10b500 0x7f728c19ca20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 --2- 172.21.15.39:0/3968277862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f728c105bc0 0x7f728c19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 >> 172.21.15.39:0/3968277862 conn(0x7f728c101610 msgr2=0x7f728c103a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 shutdown_connections 2024-09-21T01:06:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:09.629+0000 7f728be00640 1 -- 172.21.15.39:0/3968277862 wait complete. 2024-09-21T01:06:09.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:09 smithi150 ceph-mon[34945]: pgmap v468: 129 pgs: 129 active+clean; 58 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 3.0 MiB/s wr, 241 op/s 2024-09-21T01:06:10.103 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 13.29 MB/sec warmup 37 sec latency 805.516 ms 2024-09-21T01:06:10.875 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a7e3cae4', quiesce took 5.3 seconds. Will release after: 22.5 2024-09-21T01:06:11.103 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 12.94 MB/sec warmup 38 sec latency 1805.564 ms 2024-09-21T01:06:11.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:11 smithi039 ceph-mon[30415]: pgmap v469: 129 pgs: 129 active+clean; 61 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-09-21T01:06:11.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:11 smithi160 ceph-mon[34696]: pgmap v469: 129 pgs: 129 active+clean; 61 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-09-21T01:06:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:11 smithi150 ceph-mon[34945]: pgmap v469: 129 pgs: 129 active+clean; 61 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-09-21T01:06:12.103 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 12.61 MB/sec warmup 39 sec latency 2805.606 ms 2024-09-21T01:06:13.103 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 12.29 MB/sec warmup 40 sec latency 3805.688 ms 2024-09-21T01:06:13.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:13 smithi160 ceph-mon[34696]: pgmap v470: 129 pgs: 129 active+clean; 65 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 232 op/s 2024-09-21T01:06:13.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:13 smithi039 ceph-mon[30415]: pgmap v470: 129 pgs: 129 active+clean; 65 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 232 op/s 2024-09-21T01:06:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:13 smithi150 ceph-mon[34945]: pgmap v470: 129 pgs: 129 active+clean; 65 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 232 op/s 2024-09-21T01:06:14.103 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 11.99 MB/sec warmup 41 sec latency 4805.800 ms 2024-09-21T01:06:15.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 11.71 MB/sec warmup 42 sec latency 5805.866 ms 2024-09-21T01:06:15.395 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:15 smithi039 ceph-mon[30415]: pgmap v471: 129 pgs: 129 active+clean; 69 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 217 op/s 2024-09-21T01:06:15.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:15 smithi160 ceph-mon[34696]: pgmap v471: 129 pgs: 129 active+clean; 69 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 217 op/s 2024-09-21T01:06:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:15 smithi150 ceph-mon[34945]: pgmap v471: 129 pgs: 129 active+clean; 69 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 217 op/s 2024-09-21T01:06:16.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 11.44 MB/sec warmup 43 sec latency 6805.925 ms 2024-09-21T01:06:17.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 11.18 MB/sec warmup 44 sec latency 7806.018 ms 2024-09-21T01:06:17.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:17 smithi160 ceph-mon[34696]: pgmap v472: 129 pgs: 129 active+clean; 69 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 183 op/s 2024-09-21T01:06:17.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:17 smithi150 ceph-mon[34945]: pgmap v472: 129 pgs: 129 active+clean; 69 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 183 op/s 2024-09-21T01:06:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:17 smithi039 ceph-mon[30415]: pgmap v472: 129 pgs: 129 active+clean; 69 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 183 op/s 2024-09-21T01:06:18.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 10.93 MB/sec warmup 45 sec latency 8806.160 ms 2024-09-21T01:06:19.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 10.69 MB/sec warmup 46 sec latency 9806.249 ms 2024-09-21T01:06:19.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:19 smithi160 ceph-mon[34696]: pgmap v473: 129 pgs: 129 active+clean; 71 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 187 op/s 2024-09-21T01:06:19.611 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:19 smithi039 ceph-mon[30415]: pgmap v473: 129 pgs: 129 active+clean; 71 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 187 op/s 2024-09-21T01:06:19.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:19 smithi150 ceph-mon[34945]: pgmap v473: 129 pgs: 129 active+clean; 71 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 187 op/s 2024-09-21T01:06:20.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 10.46 MB/sec warmup 47 sec latency 10806.321 ms 2024-09-21T01:06:21.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 10.25 MB/sec warmup 48 sec latency 11806.364 ms 2024-09-21T01:06:21.571 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:21 smithi039 ceph-mon[30415]: pgmap v474: 129 pgs: 129 active+clean; 73 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.5 MiB/s wr, 115 op/s 2024-09-21T01:06:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:21 smithi150 ceph-mon[34945]: pgmap v474: 129 pgs: 129 active+clean; 73 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.5 MiB/s wr, 115 op/s 2024-09-21T01:06:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:21 smithi160 ceph-mon[34696]: pgmap v474: 129 pgs: 129 active+clean; 73 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.5 MiB/s wr, 115 op/s 2024-09-21T01:06:22.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 10.04 MB/sec warmup 49 sec latency 12806.406 ms 2024-09-21T01:06:23.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 9.84 MB/sec warmup 50 sec latency 13806.521 ms 2024-09-21T01:06:23.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:23 smithi039 ceph-mon[30415]: pgmap v475: 129 pgs: 129 active+clean; 72 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.6 MiB/s wr, 127 op/s 2024-09-21T01:06:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:23 smithi150 ceph-mon[34945]: pgmap v475: 129 pgs: 129 active+clean; 72 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.6 MiB/s wr, 127 op/s 2024-09-21T01:06:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:23 smithi160 ceph-mon[34696]: pgmap v475: 129 pgs: 129 active+clean; 72 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.6 MiB/s wr, 127 op/s 2024-09-21T01:06:24.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 9.64 MB/sec warmup 51 sec latency 14806.667 ms 2024-09-21T01:06:25.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 9.46 MB/sec warmup 52 sec latency 15806.751 ms 2024-09-21T01:06:25.572 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:25 smithi039 ceph-mon[30415]: pgmap v476: 129 pgs: 129 active+clean; 70 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 2.5 MiB/s wr, 96 op/s 2024-09-21T01:06:25.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:25 smithi150 ceph-mon[34945]: pgmap v476: 129 pgs: 129 active+clean; 70 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 2.5 MiB/s wr, 96 op/s 2024-09-21T01:06:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:25 smithi160 ceph-mon[34696]: pgmap v476: 129 pgs: 129 active+clean; 70 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 2.5 MiB/s wr, 96 op/s 2024-09-21T01:06:26.104 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 9.28 MB/sec warmup 53 sec latency 16806.802 ms 2024-09-21T01:06:27.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 9.11 MB/sec warmup 54 sec latency 17806.924 ms 2024-09-21T01:06:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:27 smithi039 ceph-mon[30415]: pgmap v477: 129 pgs: 129 active+clean; 71 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.1 MiB/s wr, 79 op/s 2024-09-21T01:06:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:27 smithi150 ceph-mon[34945]: pgmap v477: 129 pgs: 129 active+clean; 71 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.1 MiB/s wr, 79 op/s 2024-09-21T01:06:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:27 smithi160 ceph-mon[34696]: pgmap v477: 129 pgs: 129 active+clean; 71 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 2.1 MiB/s wr, 79 op/s 2024-09-21T01:06:28.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.94 MB/sec warmup 55 sec latency 18807.063 ms 2024-09-21T01:06:29.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.78 MB/sec warmup 56 sec latency 19807.139 ms 2024-09-21T01:06:29.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:29 smithi039 ceph-mon[30415]: pgmap v478: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-09-21T01:06:29.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:29 smithi150 ceph-mon[34945]: pgmap v478: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-09-21T01:06:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:29 smithi160 ceph-mon[34696]: pgmap v478: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-09-21T01:06:30.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.63 MB/sec warmup 57 sec latency 20807.191 ms 2024-09-21T01:06:31.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.48 MB/sec warmup 58 sec latency 21807.237 ms 2024-09-21T01:06:31.549 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:31 smithi039 ceph-mon[30415]: pgmap v479: 129 pgs: 129 active+clean; 65 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 2.0 MiB/s wr, 86 op/s 2024-09-21T01:06:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:31 smithi150 ceph-mon[34945]: pgmap v479: 129 pgs: 129 active+clean; 65 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 2.0 MiB/s wr, 86 op/s 2024-09-21T01:06:31.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:31 smithi160 ceph-mon[34696]: pgmap v479: 129 pgs: 129 active+clean; 65 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 2.0 MiB/s wr, 86 op/s 2024-09-21T01:06:32.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.34 MB/sec warmup 59 sec latency 22807.281 ms 2024-09-21T01:06:33.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.20 MB/sec warmup 60 sec latency 23807.349 ms 2024-09-21T01:06:33.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:06:33.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:33 smithi039 ceph-mon[30415]: pgmap v480: 129 pgs: 129 active+clean; 64 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.3 MiB/s wr, 97 op/s 2024-09-21T01:06:33.787 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:06:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:33 smithi150 ceph-mon[34945]: pgmap v480: 129 pgs: 129 active+clean; 64 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.3 MiB/s wr, 97 op/s 2024-09-21T01:06:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:33 smithi160 ceph-mon[34696]: pgmap v480: 129 pgs: 129 active+clean; 64 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.3 MiB/s wr, 97 op/s 2024-09-21T01:06:34.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 8.06 MB/sec warmup 61 sec latency 24807.396 ms 2024-09-21T01:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.480+0000 7f8606a00640 1 -- 172.21.15.39:0/2483576519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600075730 msgr2=0x7f8600075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.480+0000 7f8606a00640 1 --2- 172.21.15.39:0/2483576519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600075730 0x7f8600075b40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009900 tx=0x7f85f002a410 comp rx=0 tx=0).stop 2024-09-21T01:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.481+0000 7f8606a00640 1 -- 172.21.15.39:0/2483576519 shutdown_connections 2024-09-21T01:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.481+0000 7f8606a00640 1 --2- 172.21.15.39:0/2483576519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600110140 0x7f8600112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.481+0000 7f8606a00640 1 --2- 172.21.15.39:0/2483576519 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f860010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.481+0000 7f8606a00640 1 --2- 172.21.15.39:0/2483576519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600075730 0x7f8600075b40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.481+0000 7f8606a00640 1 -- 172.21.15.39:0/2483576519 >> 172.21.15.39:0/2483576519 conn(0x7f860007b850 msgr2=0x7f860007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.481+0000 7f8606a00640 1 -- 172.21.15.39:0/2483576519 shutdown_connections 2024-09-21T01:06:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.482+0000 7f8606a00640 1 -- 172.21.15.39:0/2483576519 wait complete. 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.483+0000 7f8606a00640 1 Processor -- start 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 -- start start 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600075730 0x7f8600104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f8600104fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600110140 0x7f860010f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8600115c80 con 0x7f8600075730 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8600115b00 con 0x7f8600110140 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.484+0000 7f8606a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8600115e00 con 0x7f8600076080 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.485+0000 7f85ff400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f8600104fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.485+0000 7f85ff400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f8600104fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48378/0 (socket says 172.21.15.39:48378) 2024-09-21T01:06:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.485+0000 7f85ff400640 1 -- 172.21.15.39:0/1220196130 learned_addr learned my addr 172.21.15.39:0/1220196130 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.485+0000 7f85ffe00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600075730 0x7f8600104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.485+0000 7f8604a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600110140 0x7f860010f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:34.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.486+0000 7f85ff400640 1 -- 172.21.15.39:0/1220196130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600110140 msgr2=0x7f860010f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:34.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.486+0000 7f85ff400640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600110140 0x7f860010f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.486+0000 7f85ff400640 1 -- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600075730 msgr2=0x7f8600104a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:34.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.487+0000 7f85ff400640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600075730 0x7f8600104a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.487+0000 7f85ff400640 1 -- 172.21.15.39:0/1220196130 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860010fb50 con 0x7f8600076080 2024-09-21T01:06:34.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.488+0000 7f8604a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600110140 0x7f860010f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:06:34.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.488+0000 7f85ffe00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600075730 0x7f8600104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:06:34.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.490+0000 7f85ff400640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f8600104fd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00e9f0 tx=0x7f85ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:34.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.490+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec00ccc0 con 0x7f8600076080 2024-09-21T01:06:34.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.490+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85ec0108e0 con 0x7f8600076080 2024-09-21T01:06:34.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.490+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860010fde0 con 0x7f8600076080 2024-09-21T01:06:34.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.490+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860010a4b0 con 0x7f8600076080 2024-09-21T01:06:34.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.490+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec018860 con 0x7f8600076080 2024-09-21T01:06:34.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.494+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e00052c0 con 0x7f8600076080 2024-09-21T01:06:34.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.500+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f85ec010430 con 0x7f8600076080 2024-09-21T01:06:34.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.506+0000 7f85fcc00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85e40b6700 0x7f85e40b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:34.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.507+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f85ec0e7d60 con 0x7f8600076080 2024-09-21T01:06:34.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.507+0000 7f85ffe00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85e40b6700 0x7f85e40b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:34.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.509+0000 7f85ffe00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85e40b6700 0x7f85e40b8c10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f85f0000c00 tx=0x7f85f0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:34.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.510+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f85ec0aded0 con 0x7f8600076080 2024-09-21T01:06:34.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.780+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85e0005880 con 0x7f8600076080 2024-09-21T01:06:34.783 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:06:34.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.781+0000 7f85fcc00640 1 -- 172.21.15.39:0/1220196130 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11914 (secure 0 0 0) 0x7f85ec0b22b0 con 0x7f8600076080 2024-09-21T01:06:34.786 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"btime":"2024-09-21T01:05:55:033931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:05:54.497271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:06:34.787 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-09-21T01:06:34.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85e40b6700 msgr2=0x7f85e40b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85e40b6700 0x7f85e40b8c10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f85f0000c00 tx=0x7f85f0031400 comp rx=0 tx=0).stop 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 msgr2=0x7f8600104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f8600104fd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00e9f0 tx=0x7f85ec00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 shutdown_connections 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85e40b6700 0x7f85e40b8c10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8600110140 0x7f860010f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8600076080 0x7f8600104fd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 --2- 172.21.15.39:0/1220196130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8600075730 0x7f8600104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.786+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 >> 172.21.15.39:0/1220196130 conn(0x7f860007b850 msgr2=0x7f86001017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:34.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.787+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 shutdown_connections 2024-09-21T01:06:34.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:34.787+0000 7f8606a00640 1 -- 172.21.15.39:0/1220196130 wait complete. 2024-09-21T01:06:35.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 7.93 MB/sec warmup 62 sec latency 25807.441 ms 2024-09-21T01:06:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:35 smithi039 ceph-mon[30415]: pgmap v481: 129 pgs: 129 active+clean; 65 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-21T01:06:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:35 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1220196130' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:35 smithi150 ceph-mon[34945]: pgmap v481: 129 pgs: 129 active+clean; 65 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-21T01:06:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:35 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1220196130' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:35 smithi160 ceph-mon[34696]: pgmap v481: 129 pgs: 129 active+clean; 65 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.2 MiB/s wr, 92 op/s 2024-09-21T01:06:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:35 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1220196130' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:36.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 7.81 MB/sec warmup 63 sec latency 26807.484 ms 2024-09-21T01:06:36.452 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id a7e3cae4 --release --await' 2024-09-21T01:06:36.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id a7e3cae4 --release --await 2024-09-21T01:06:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:06:37.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88078 7.68 MB/sec warmup 64 sec latency 27807.547 ms 2024-09-21T01:06:37.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 -- 172.21.15.39:0/3192202804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 msgr2=0x7fd6dc0757b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/3192202804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc0757b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009900 tx=0x7fd6d002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 -- 172.21.15.39:0/3192202804 shutdown_connections 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/3192202804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 0x7fd6dc116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/3192202804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd6dc075dd0 0x7fd6dc0761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/3192202804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc0757b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 -- 172.21.15.39:0/3192202804 >> 172.21.15.39:0/3192202804 conn(0x7fd6dc0ff440 msgr2=0x7fd6dc101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 -- 172.21.15.39:0/3192202804 shutdown_connections 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.635+0000 7fd6e2e00640 1 -- 172.21.15.39:0/3192202804 wait complete. 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.636+0000 7fd6e2e00640 1 Processor -- start 2024-09-21T01:06:37.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 -- start start 2024-09-21T01:06:37.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd6dc075dd0 0x7fd6dc1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc1a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 0x7fd6dc1a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc119ec0 con 0x7fd6dc077380 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc119d40 con 0x7fd6dc114280 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc11a040 con 0x7fd6dc075dd0 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc1a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e1a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd6dc075dd0 0x7fd6dc1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc1a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35330/0 (socket says 172.21.15.39:35330) 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e1000640 1 -- 172.21.15.39:0/297691378 learned_addr learned my addr 172.21.15.39:0/297691378 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.637+0000 7fd6e2400640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 0x7fd6dc1a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.638+0000 7fd6e2400640 1 -- 172.21.15.39:0/297691378 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd6dc075dd0 msgr2=0x7fd6dc1a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.638+0000 7fd6e2400640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd6dc075dd0 0x7fd6dc1a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.638+0000 7fd6e2400640 1 -- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 msgr2=0x7fd6dc1a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.638+0000 7fd6e2400640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc1a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.638+0000 7fd6e2400640 1 -- 172.21.15.39:0/297691378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6dc1a6880 con 0x7fd6dc114280 2024-09-21T01:06:37.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.638+0000 7fd6e2400640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 0x7fd6dc1a6150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009fd0 tx=0x7fd6d400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:37.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.639+0000 7fd6da000640 1 -- 172.21.15.39:0/297691378 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400ce20 con 0x7fd6dc114280 2024-09-21T01:06:37.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.639+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6dc1a6b70 con 0x7fd6dc114280 2024-09-21T01:06:37.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.639+0000 7fd6da000640 1 -- 172.21.15.39:0/297691378 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d4005020 con 0x7fd6dc114280 2024-09-21T01:06:37.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.639+0000 7fd6da000640 1 -- 172.21.15.39:0/297691378 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4010690 con 0x7fd6dc114280 2024-09-21T01:06:37.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.639+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6dc1ad2d0 con 0x7fd6dc114280 2024-09-21T01:06:37.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.643+0000 7fd6eafef740 1 Processor -- start 2024-09-21T01:06:37.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6da000640 1 -- 172.21.15.39:0/297691378 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fd6d40054d0 con 0x7fd6dc114280 2024-09-21T01:06:37.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 -- start start 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba2399c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba2399c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba2399cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ba23913290 con 0x55ba235cc660 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55ba239026c0 con 0x55ba235d4840 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6eafef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ba239064d0 con 0x55ba235f5520 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.644+0000 7fd6e1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba2399c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba2399c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35342/0 (socket says 172.21.15.39:35342) 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 -- 172.21.15.39:0/1423580946 learned_addr learned my addr 172.21.15.39:0/1423580946 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:37.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba2399c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.647 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:37 smithi039 ceph-mon[30415]: pgmap v482: 129 pgs: 129 active+clean; 66 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1000640 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba2399cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 -- 172.21.15.39:0/1423580946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 msgr2=0x55ba2399c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba2399c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 -- 172.21.15.39:0/1423580946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 msgr2=0x55ba2399cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba2399cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1a00640 1 -- 172.21.15.39:0/1423580946 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba2388f460 con 0x55ba235cc660 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6e1000640 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba2399cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.645+0000 7fd6da000640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd6c40b6700 0x7fd6c40b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.646+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba2399c880 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009900 tx=0x7fd6d0032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2641e705c84542a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.646+0000 7fd6da000640 1 -- 172.21.15.39:0/297691378 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd6d40e8160 con 0x7fd6dc114280 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.646+0000 7fd6e1000640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd6c40b6700 0x7fd6c40b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.646+0000 7fd6cea00640 1 -- 172.21.15.39:0/1423580946 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0040070 con 0x55ba235cc660 2024-09-21T01:06:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.646+0000 7fd6cea00640 1 -- 172.21.15.39:0/1423580946 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d0004030 con 0x55ba235cc660 2024-09-21T01:06:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.646+0000 7fd6cea00640 1 -- 172.21.15.39:0/1423580946 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0035040 con 0x55ba235cc660 2024-09-21T01:06:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 -- 172.21.15.39:0/1423580946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 msgr2=0x55ba2399c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6e1000640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd6c40b6700 0x7fd6c40b8c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80095e0 tx=0x7fd6c8013040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba2399c880 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009900 tx=0x7fd6d0032b90 comp rx=0 tx=0).stop 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 -- 172.21.15.39:0/1423580946 shutdown_connections 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba2399cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba2399c880 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1423580946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba2399c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.647+0000 7fd6eafef740 1 -- 172.21.15.39:0/1423580946 >> 172.21.15.39:0/1423580946 conn(0x55ba235e9910 msgr2=0x55ba2399a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.648+0000 7fd6eafef740 1 -- 172.21.15.39:0/1423580946 shutdown_connections 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.648+0000 7fd6eafef740 1 -- 172.21.15.39:0/1423580946 wait complete. 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.648+0000 7fd6eafef740 1 -- start start 2024-09-21T01:06:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.648+0000 7fd6eafef740 1 Processor -- start 2024-09-21T01:06:37.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba239a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba239a8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba239b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ba23763bb0 con 0x55ba235cc660 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55ba2389d6c0 con 0x55ba235d4840 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ba239aaac0 con 0x55ba235f5520 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6eafef740 10 client.0 _ll_register_callbacks cb 0x55ba238b8490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6e2400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba239b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6e2400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba239b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48410/0 (socket says 172.21.15.39:48410) 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6e2400640 1 -- 172.21.15.39:0/1706316834 learned_addr learned my addr 172.21.15.39:0/1706316834 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6e1000640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba239a8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.649+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba239a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6e2400640 1 -- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 msgr2=0x55ba239a8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6e2400640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba239a8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6e2400640 1 -- 172.21.15.39:0/1706316834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 msgr2=0x55ba239a7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6e2400640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba239a7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6e2400640 1 -- 172.21.15.39:0/1706316834 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba23660b00 con 0x55ba235f5520 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6e2400640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba239b3de0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6d401a040 tx=0x7fd6d40ad420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=744b2d6a95fa5fd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6c3400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.650+0000 7fd6c3400640 1 -- 172.21.15.39:0/1706316834 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40b2020 con 0x55ba235f5520 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6c3400640 1 -- 172.21.15.39:0/1706316834 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd6d40b7160 con 0x55ba235f5520 2024-09-21T01:06:37.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6d0004030 con 0x55ba235f5520 2024-09-21T01:06:37.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6c3400640 1 -- 172.21.15.39:0/1706316834 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6d40b26b0 con 0x55ba235f5520 2024-09-21T01:06:37.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 10 client.25639 fetch_fsmap learned FSMap version 28 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ba236d7960 con 0x55ba235f5520 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6c3400640 1 -- 172.21.15.39:0/1706316834 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d0004030 con 0x55ba235f5520 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6c3400640 1 -- 172.21.15.39:0/1706316834 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd6d40b22e0 con 0x55ba235f5520 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 10 client.25639 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 10 client.25639 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 20 client.25639 populate_metadata read hostname 'smithi039' 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.651+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba235b3ea0 0x55ba239b9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.652+0000 7fd6eafef740 4 client.25639 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.652+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba239151e0 con 0x55ba235b3ea0 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.652+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba235b3ea0 0x55ba239b9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.653+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba235b3ea0 0x55ba239b9e60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2dae16e34a480b0 server_cookie=e439d869127a0597 in_seq=0 out_seq=0 2024-09-21T01:06:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.653+0000 7fd6c3400640 10 client.25639 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:37.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.653+0000 7fd6c3400640 1 -- 172.21.15.39:0/1706316834 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ba239151e0 con 0x55ba235b3ea0 2024-09-21T01:06:37.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.653+0000 7fd6c3400640 10 client.25639 handle_command_reply: tid=0 2024-09-21T01:06:37.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.653+0000 7fd6eafef740 1 client.25639 shutdown 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 msgr2=0x55ba239b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba239b3de0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6d401a040 tx=0x7fd6d40ad420 comp rx=0 tx=0).stop 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 shutdown_connections 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba235b3ea0 0x55ba239b9e60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba239b3de0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235d4840 0x55ba239a8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 --2- 172.21.15.39:0/1706316834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235cc660 0x55ba239a7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 >> 172.21.15.39:0/1706316834 conn(0x55ba235e9910 msgr2=0x55ba2399f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6c3400640 0 client.25639 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 shutdown_connections 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 1 -- 172.21.15.39:0/1706316834 wait complete. 2024-09-21T01:06:37.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.654+0000 7fd6eafef740 20 client.25639 trim_cache size 0 max 16384 2024-09-21T01:06:37.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.672+0000 7fd6eafef740 1 Processor -- start 2024-09-21T01:06:37.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.672+0000 7fd6eafef740 1 -- start start 2024-09-21T01:06:37.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba2399fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235d4840 0x55ba239a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235f5520 0x55ba239a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6eafef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc1b9b10 con 0x55ba235f5520 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6eafef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc1b73e0 con 0x55ba235cc660 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6eafef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc1b5230 con 0x55ba235d4840 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6e1a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba2399fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6e2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235f5520 0x55ba239a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6e1a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba2399fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38282/0 (socket says 172.21.15.39:38282) 2024-09-21T01:06:37.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6e1a00640 1 -- 172.21.15.39:0/3313362221 learned_addr learned my addr 172.21.15.39:0/3313362221 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:37.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.673+0000 7fd6e1000640 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235d4840 0x55ba239a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e1a00640 1 -- 172.21.15.39:0/3313362221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235d4840 msgr2=0x55ba239a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235d4840 0x55ba239a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e1a00640 1 -- 172.21.15.39:0/3313362221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235f5520 msgr2=0x55ba239a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235f5520 0x55ba239a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e1a00640 1 -- 172.21.15.39:0/3313362221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba239a3630 con 0x55ba235cc660 2024-09-21T01:06:37.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e2400640 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235f5520 0x55ba239a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:06:37.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba2399fb50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55ba2399afc0 tx=0x7fd6d0005180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce1cd4c45538ad5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:37.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6ce000640 1 -- 172.21.15.39:0/3313362221 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0035040 con 0x55ba235cc660 2024-09-21T01:06:37.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6ce000640 1 -- 172.21.15.39:0/3313362221 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d0004460 con 0x55ba235cc660 2024-09-21T01:06:37.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.674+0000 7fd6ce000640 1 -- 172.21.15.39:0/3313362221 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0040040 con 0x55ba235cc660 2024-09-21T01:06:37.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 -- 172.21.15.39:0/3313362221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 msgr2=0x55ba2399fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba2399fb50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55ba2399afc0 tx=0x7fd6d0005180 comp rx=0 tx=0).stop 2024-09-21T01:06:37.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 -- 172.21.15.39:0/3313362221 shutdown_connections 2024-09-21T01:06:37.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235f5520 0x55ba239a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235d4840 0x55ba239a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3313362221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba2399fb50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 -- 172.21.15.39:0/3313362221 >> 172.21.15.39:0/3313362221 conn(0x55ba235e9910 msgr2=0x55ba2399f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:37.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 -- 172.21.15.39:0/3313362221 shutdown_connections 2024-09-21T01:06:37.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.675+0000 7fd6eafef740 1 -- 172.21.15.39:0/3313362221 wait complete. 2024-09-21T01:06:37.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.676+0000 7fd6eafef740 1 -- start start 2024-09-21T01:06:37.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.676+0000 7fd6eafef740 1 Processor -- start 2024-09-21T01:06:37.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.676+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba239ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 0x55ba239acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6eafef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba23a1d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6eafef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc1b58a0 con 0x55ba235d4840 2024-09-21T01:06:37.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6eafef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc1b6d10 con 0x55ba235cc660 2024-09-21T01:06:37.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6eafef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc1b5f70 con 0x55ba235f5520 2024-09-21T01:06:37.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6eafef740 10 client.0 _ll_register_callbacks cb 0x7fd6d40aa2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:06:37.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 0x55ba239acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 0x55ba239acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35384/0 (socket says 172.21.15.39:35384) 2024-09-21T01:06:37.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 -- 172.21.15.39:0/3275201628 learned_addr learned my addr 172.21.15.39:0/3275201628 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:37.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba239ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e2400640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba23a1d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 -- 172.21.15.39:0/3275201628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 msgr2=0x55ba23a1d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba23a1d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 -- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 msgr2=0x55ba239ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:37.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba239ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.677+0000 7fd6e1000640 1 -- 172.21.15.39:0/3275201628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba239a6d00 con 0x55ba235d4840 2024-09-21T01:06:37.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6e1000640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 0x55ba239acbb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800fcb0 tx=0x7fd6c8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2fb8190f03c9b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6c3400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6c3400640 1 -- 172.21.15.39:0/3275201628 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8005020 con 0x55ba235d4840 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6d0004460 con 0x55ba235d4840 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6c3400640 1 -- 172.21.15.39:0/3275201628 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd6c801a070 con 0x55ba235d4840 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6c3400640 1 -- 172.21.15.39:0/3275201628 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6c8035050 con 0x55ba235d4840 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6eafef740 10 client.15624 fetch_fsmap learned FSMap version 28 2024-09-21T01:06:37.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.678+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ba23a250d0 con 0x55ba235d4840 2024-09-21T01:06:37.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6c3400640 1 -- 172.21.15.39:0/3275201628 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d0004460 con 0x55ba235d4840 2024-09-21T01:06:37.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6c3400640 1 -- 172.21.15.39:0/3275201628 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd6c801f440 con 0x55ba235d4840 2024-09-21T01:06:37.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6eafef740 10 client.15624 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:06:37.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6eafef740 10 client.15624 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:06:37.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6eafef740 20 client.15624 populate_metadata read hostname 'smithi039' 2024-09-21T01:06:37.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba23589250 0x55ba23a27970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:37.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6eafef740 4 client.15624 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a7e3cae4", "release": true, "await": true}] 2024-09-21T01:06:37.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.679+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a7e3cae4", "release": true, "await": true}) -- 0x55ba23910c90 con 0x55ba23589250 2024-09-21T01:06:37.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.680+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba23589250 0x55ba23a27970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:37.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.680+0000 7fd6e1a00640 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba23589250 0x55ba23a27970 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d89e44b19c1fa5 server_cookie=e5e314dba75417e4 in_seq=0 out_seq=0 2024-09-21T01:06:37.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.680+0000 7fd6c3400640 10 client.15624 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:37.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.686+0000 7fd6c3400640 1 -- 172.21.15.39:0/3275201628 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ba23910c90 con 0x55ba23589250 2024-09-21T01:06:37.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.686+0000 7fd6c3400640 10 client.15624 handle_command_reply: tid=0 2024-09-21T01:06:37.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.686+0000 7fd6eafef740 1 client.15624 shutdown 2024-09-21T01:06:37.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 msgr2=0x55ba239acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 0x55ba239acbb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800fcb0 tx=0x7fd6c8005f90 comp rx=0 tx=0).stop 2024-09-21T01:06:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 shutdown_connections 2024-09-21T01:06:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ba23589250 0x55ba23a27970 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ba235f5520 0x55ba23a1d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6c3400640 0 client.15624 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ba235d4840 0x55ba239acbb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 --2- 172.21.15.39:0/3275201628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ba235cc660 0x55ba239ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 >> 172.21.15.39:0/3275201628 conn(0x55ba235e9910 msgr2=0x55ba2399d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:37.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 shutdown_connections 2024-09-21T01:06:37.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 1 -- 172.21.15.39:0/3275201628 wait complete. 2024-09-21T01:06:37.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.687+0000 7fd6eafef740 20 client.15624 trim_cache size 0 max 16384 2024-09-21T01:06:37.696 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:06:37.696 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 28, 2024-09-21T01:06:37.696 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:06:37.696 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 12, 2024-09-21T01:06:37.696 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:06:37.697 INFO:teuthology.orchestra.run.smithi039.stdout: "a7e3cae4": { 2024-09-21T01:06:37.697 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 12, 2024-09-21T01:06:37.697 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:06:37.697 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:06:37.697 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:06:37.697 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 147.8, 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:37.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:37.700 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:06:37.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd6c40b6700 msgr2=0x7fd6c40b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd6c40b6700 0x7fd6c40b8c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80095e0 tx=0x7fd6c8013040 comp rx=0 tx=0).stop 2024-09-21T01:06:37.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 msgr2=0x7fd6dc1a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:37.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 0x7fd6dc1a6150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009fd0 tx=0x7fd6d400ef10 comp rx=0 tx=0).stop 2024-09-21T01:06:37.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 shutdown_connections 2024-09-21T01:06:37.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd6c40b6700 0x7fd6c40b8c10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6dc114280 0x7fd6dc1a6150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6dc077380 0x7fd6dc1a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 --2- 172.21.15.39:0/297691378 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd6dc075dd0 0x7fd6dc1a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:37.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 >> 172.21.15.39:0/297691378 conn(0x7fd6dc0ff440 msgr2=0x7fd6dc110180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:37.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 shutdown_connections 2024-09-21T01:06:37.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:37.688+0000 7fd6e2e00640 1 -- 172.21.15.39:0/297691378 wait complete. 2024-09-21T01:06:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:37 smithi150 ceph-mon[34945]: pgmap v482: 129 pgs: 129 active+clean; 66 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-09-21T01:06:37.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:37 smithi160 ceph-mon[34696]: pgmap v482: 129 pgs: 129 active+clean; 66 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-09-21T01:06:38.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 88844 7.66 MB/sec warmup 65 sec latency 28391.245 ms 2024-09-21T01:06:38.688 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a7e3cae4', total seconds elapsed: 33.1 2024-09-21T01:06:38.688 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.8 2024-09-21T01:06:38.688 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:06:39.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 90348 7.64 MB/sec warmup 66 sec latency 18.025 ms 2024-09-21T01:06:39.110 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:06:39.557 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:39 smithi039 ceph-mon[30415]: pgmap v483: 129 pgs: 129 active+clean; 63 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 2.0 MiB/s wr, 90 op/s 2024-09-21T01:06:39.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:39 smithi150 ceph-mon[34945]: pgmap v483: 129 pgs: 129 active+clean; 63 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 2.0 MiB/s wr, 90 op/s 2024-09-21T01:06:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:39 smithi160 ceph-mon[34696]: pgmap v483: 129 pgs: 129 active+clean; 63 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 2.0 MiB/s wr, 90 op/s 2024-09-21T01:06:39.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 -- 172.21.15.39:0/1674407488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 msgr2=0x7fde48112850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:39.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1674407488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 0x7fde48112850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fde40009ca0 tx=0x7fde4002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 -- 172.21.15.39:0/1674407488 shutdown_connections 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1674407488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 0x7fde48112850 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1674407488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6d0 0x7fde4810bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1674407488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde481061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 -- 172.21.15.39:0/1674407488 >> 172.21.15.39:0/1674407488 conn(0x7fde481017c0 msgr2=0x7fde48103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 -- 172.21.15.39:0/1674407488 shutdown_connections 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.889+0000 7fde4fc00640 1 -- 172.21.15.39:0/1674407488 wait complete. 2024-09-21T01:06:39.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.890+0000 7fde4fc00640 1 Processor -- start 2024-09-21T01:06:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.890+0000 7fde4fc00640 1 -- start start 2024-09-21T01:06:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4fc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde4819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:39.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6d0 0x7fde4819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4fc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 0x7fde481a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fde48115d30 con 0x7fde4810b6d0 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4fc00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde48115bb0 con 0x7fde48105d90 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4fc00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fde48115eb0 con 0x7fde4810c020 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4d200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde4819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4d200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde4819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38304/0 (socket says 172.21.15.39:38304) 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4d200640 1 -- 172.21.15.39:0/1768603704 learned_addr learned my addr 172.21.15.39:0/1768603704 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde3fe00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6d0 0x7fde4819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:39.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.891+0000 7fde4dc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 0x7fde481a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4d200640 1 -- 172.21.15.39:0/1768603704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 msgr2=0x7fde481a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4d200640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 0x7fde481a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4d200640 1 -- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6d0 msgr2=0x7fde4819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4d200640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6d0 0x7fde4819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4d200640 1 -- 172.21.15.39:0/1768603704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde481a25f0 con 0x7fde48105d90 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4d200640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde4819c680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e9f0 tx=0x7fde3800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3800cd80 con 0x7fde48105d90 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde380054d0 con 0x7fde48105d90 2024-09-21T01:06:39.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38010690 con 0x7fde48105d90 2024-09-21T01:06:39.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde481a28e0 con 0x7fde48105d90 2024-09-21T01:06:39.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.892+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde481a9040 con 0x7fde48105d90 2024-09-21T01:06:39.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.894+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde481061a0 con 0x7fde48105d90 2024-09-21T01:06:39.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.896+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fde38004e90 con 0x7fde48105d90 2024-09-21T01:06:39.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.897+0000 7fde46000640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde2c0b6700 0x7fde2c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:39.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.898+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fde380e8180 con 0x7fde48105d90 2024-09-21T01:06:39.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.898+0000 7fde3fe00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde2c0b6700 0x7fde2c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:39.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.898+0000 7fde3fe00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde2c0b6700 0x7fde2c0b8c10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fde48104fa0 tx=0x7fde30005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:39.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:39.899+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fde380ae2f0 con 0x7fde48105d90 2024-09-21T01:06:40.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 92670 7.73 MB/sec warmup 67 sec latency 31.030 ms 2024-09-21T01:06:40.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.186+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde4810de70 con 0x7fde48105d90 2024-09-21T01:06:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.192+0000 7fde46000640 1 -- 172.21.15.39:0/1768603704 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11914 (secure 0 0 0) 0x7fde380b26d0 con 0x7fde48105d90 2024-09-21T01:06:40.194 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:06:40.194 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"btime":"2024-09-21T01:05:55:033931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:05:54.497271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:06:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.195+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde2c0b6700 msgr2=0x7fde2c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.195+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde2c0b6700 0x7fde2c0b8c10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fde48104fa0 tx=0x7fde30005f90 comp rx=0 tx=0).stop 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 msgr2=0x7fde4819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde4819c680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e9f0 tx=0x7fde3800edf0 comp rx=0 tx=0).stop 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 shutdown_connections 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde4810c020 0x7fde481a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4810b6d0 0x7fde4819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde2c0b6700 0x7fde2c0b8c10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 --2- 172.21.15.39:0/1768603704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde48105d90 0x7fde4819c680 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 >> 172.21.15.39:0/1768603704 conn(0x7fde481017c0 msgr2=0x7fde48108be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 shutdown_connections 2024-09-21T01:06:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:40.196+0000 7fde4fc00640 1 -- 172.21.15.39:0/1768603704 wait complete. 2024-09-21T01:06:40.676 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:40 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1768603704' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:40.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:40 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1768603704' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:40.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:40 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1768603704' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:06:41.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 92671 7.62 MB/sec warmup 68 sec latency 951.126 ms 2024-09-21T01:06:41.151 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0' 2024-09-21T01:06:41.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0 2024-09-21T01:06:41.532 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:06:41.671 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:41 smithi039 ceph-mon[30415]: pgmap v484: 129 pgs: 129 active+clean; 62 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.7 MiB/s wr, 73 op/s 2024-09-21T01:06:41.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:41 smithi150 ceph-mon[34945]: pgmap v484: 129 pgs: 129 active+clean; 62 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.7 MiB/s wr, 73 op/s 2024-09-21T01:06:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:41 smithi160 ceph-mon[34696]: pgmap v484: 129 pgs: 129 active+clean; 62 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.7 MiB/s wr, 73 op/s 2024-09-21T01:06:42.105 INFO:tasks.workunit.client.0.smithi039.stdout: 1 93329 7.54 MB/sec warmup 69 sec latency 1265.761 ms 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 -- 172.21.15.39:0/2257561424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7830105d80 msgr2=0x7f7830106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 --2- 172.21.15.39:0/2257561424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7830105d80 0x7f7830106190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7824009900 tx=0x7f782402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 -- 172.21.15.39:0/2257561424 shutdown_connections 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 --2- 172.21.15.39:0/2257561424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783010c010 0x7f7830112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 --2- 172.21.15.39:0/2257561424 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783010b6c0 0x7f783010bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 --2- 172.21.15.39:0/2257561424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7830105d80 0x7f7830106190 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 -- 172.21.15.39:0/2257561424 >> 172.21.15.39:0/2257561424 conn(0x7f78301017d0 msgr2=0x7f7830103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 -- 172.21.15.39:0/2257561424 shutdown_connections 2024-09-21T01:06:42.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.347+0000 7f7838800640 1 -- 172.21.15.39:0/2257561424 wait complete. 2024-09-21T01:06:42.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 Processor -- start 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 -- start start 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7830105d80 0x7f783007b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783010b6c0 0x7f7830077920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 0x7f7830078040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7830115cd0 con 0x7f783010c010 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7830115b50 con 0x7f7830105d80 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.349+0000 7f7838800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7830115e50 con 0x7f783010b6c0 2024-09-21T01:06:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7836800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 0x7f7830078040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7836800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 0x7f7830078040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35422/0 (socket says 172.21.15.39:35422) 2024-09-21T01:06:42.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7836800640 1 -- 172.21.15.39:0/2493774547 learned_addr learned my addr 172.21.15.39:0/2493774547 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:42.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7835e00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7830105d80 0x7f783007b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7835e00640 1 -- 172.21.15.39:0/2493774547 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783010b6c0 msgr2=0x7f7830077920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7835e00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783010b6c0 0x7f7830077920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7835e00640 1 -- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 msgr2=0x7f7830078040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7835e00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 0x7f7830078040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7835e00640 1 -- 172.21.15.39:0/2493774547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783019c3b0 con 0x7f7830105d80 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.350+0000 7f7836800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 0x7f7830078040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.351+0000 7f7835e00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7830105d80 0x7f783007b640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f782402a910 tx=0x7f7824031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.351+0000 7f782aa00640 1 -- 172.21.15.39:0/2493774547 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782403f030 con 0x7f7830105d80 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.351+0000 7f782aa00640 1 -- 172.21.15.39:0/2493774547 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7824004490 con 0x7f7830105d80 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.351+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783019c640 con 0x7f7830105d80 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.351+0000 7f782aa00640 1 -- 172.21.15.39:0/2493774547 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824035070 con 0x7f7830105d80 2024-09-21T01:06:42.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.351+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783019cb70 con 0x7f7830105d80 2024-09-21T01:06:42.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.355+0000 7f782aa00640 1 -- 172.21.15.39:0/2493774547 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7824042030 con 0x7f7830105d80 2024-09-21T01:06:42.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.355+0000 7f783f5f8740 1 Processor -- start 2024-09-21T01:06:42.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.355+0000 7f783f5f8740 1 -- start start 2024-09-21T01:06:42.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 0x5562fd667350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd667890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd29eb60 0x5562fd667dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f783f5f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5562fd5dcca0 con 0x5562fd296980 2024-09-21T01:06:42.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f783f5f8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5562fd5cc0d0 con 0x5562fd29eb60 2024-09-21T01:06:42.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f783f5f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5562fd5cfee0 con 0x5562fd2bf840 2024-09-21T01:06:42.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f7836800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 0x5562fd667350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f7836800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 0x5562fd667350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48472/0 (socket says 172.21.15.39:48472) 2024-09-21T01:06:42.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f7836800640 1 -- 172.21.15.39:0/392197354 learned_addr learned my addr 172.21.15.39:0/392197354 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:42.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f7835400640 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd29eb60 0x5562fd667dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.356+0000 7f7835e00640 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd667890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7836800640 1 -- 172.21.15.39:0/392197354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd29eb60 msgr2=0x5562fd667dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7836800640 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd29eb60 0x5562fd667dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7836800640 1 -- 172.21.15.39:0/392197354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 msgr2=0x5562fd667890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7836800640 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd667890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7836800640 1 -- 172.21.15.39:0/392197354 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562fd57e400 con 0x5562fd2bf840 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7835e00640 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd667890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f7836800640 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 0x5562fd667350 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f782c0118b0 tx=0x7f782c011d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd060e1f5d213b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:42.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f782aa00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78180b6700 0x7f78180b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.357+0000 7f781f000640 1 -- 172.21.15.39:0/392197354 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c01e070 con 0x5562fd2bf840 2024-09-21T01:06:42.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.358+0000 7f782aa00640 1 -- 172.21.15.39:0/2493774547 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7824107dc0 con 0x7f7830105d80 2024-09-21T01:06:42.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.358+0000 7f781f000640 1 -- 172.21.15.39:0/392197354 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f782c00fe70 con 0x5562fd2bf840 2024-09-21T01:06:42.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.358+0000 7f781f000640 1 -- 172.21.15.39:0/392197354 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c010b20 con 0x5562fd2bf840 2024-09-21T01:06:42.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.358+0000 7f7835e00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78180b6700 0x7f78180b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f7835e00640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78180b6700 0x7f78180b8c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5562fd665fd0 tx=0x7f7824000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 -- 172.21.15.39:0/392197354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 msgr2=0x5562fd667350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 0x5562fd667350 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f782c0118b0 tx=0x7f782c011d70 comp rx=0 tx=0).stop 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 -- 172.21.15.39:0/392197354 shutdown_connections 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd29eb60 0x5562fd667dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd667890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 --2- 172.21.15.39:0/392197354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd2bf840 0x5562fd667350 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 -- 172.21.15.39:0/392197354 >> 172.21.15.39:0/392197354 conn(0x5562fd2b3990 msgr2=0x5562fd665240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 -- 172.21.15.39:0/392197354 shutdown_connections 2024-09-21T01:06:42.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.359+0000 7f783f5f8740 1 -- 172.21.15.39:0/392197354 wait complete. 2024-09-21T01:06:42.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.360+0000 7f783f5f8740 1 -- start start 2024-09-21T01:06:42.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.360+0000 7f783f5f8740 1 Processor -- start 2024-09-21T01:06:42.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd672970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd672eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd6733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5562fd42d7a0 con 0x5562fd29eb60 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5562fd571a10 con 0x5562fd2bf840 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5562fd675b30 con 0x5562fd296980 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f783f5f8740 10 client.0 _ll_register_callbacks cb 0x5562fd581be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f7836800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd672970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f7836800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd672970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48488/0 (socket says 172.21.15.39:48488) 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f7836800640 1 -- 172.21.15.39:0/1072834673 learned_addr learned my addr 172.21.15.39:0/1072834673 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f7836800640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd6733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.361+0000 7f7835400640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd672eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7836800640 1 -- 172.21.15.39:0/1072834673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 msgr2=0x5562fd672970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7836800640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd672970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7836800640 1 -- 172.21.15.39:0/1072834673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 msgr2=0x5562fd672eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7836800640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd672eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7836800640 1 -- 172.21.15.39:0/1072834673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562fd339b90 con 0x5562fd2bf840 2024-09-21T01:06:42.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7836800640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd6733f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5562fd6697f0 tx=0x7f782c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89b1ed2559cb1620 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7813400640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.362+0000 7f7813400640 1 -- 172.21.15.39:0/1072834673 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c01e070 con 0x5562fd2bf840 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f7813400640 1 -- 172.21.15.39:0/1072834673 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f782c010860 con 0x5562fd2bf840 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f782c00fe70 con 0x5562fd2bf840 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f7813400640 1 -- 172.21.15.39:0/1072834673 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f782c010a60 con 0x5562fd2bf840 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 10 client.25670 fetch_fsmap learned FSMap version 28 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562fd6816e0 con 0x5562fd2bf840 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f7813400640 1 -- 172.21.15.39:0/1072834673 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f782c00fe70 con 0x5562fd2bf840 2024-09-21T01:06:42.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f7813400640 1 -- 172.21.15.39:0/1072834673 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f782c005580 con 0x5562fd2bf840 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 10 client.25670 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 10 client.25670 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 20 client.25670 populate_metadata read hostname 'smithi039' 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.363+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd27e1c0 0x5562fd684b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.364+0000 7f783f5f8740 4 client.25670 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.364+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562fd5debf0 con 0x5562fd27e1c0 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.364+0000 7f7835400640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd27e1c0 0x5562fd684b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.365+0000 7f7835400640 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd27e1c0 0x5562fd684b50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f41cd89d533bbaa server_cookie=ec85edbce765069 in_seq=0 out_seq=0 2024-09-21T01:06:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.365+0000 7f7813400640 10 client.25670 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:42.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f7813400640 1 -- 172.21.15.39:0/1072834673 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5562fd5debf0 con 0x5562fd27e1c0 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f7813400640 10 client.25670 handle_command_reply: tid=0 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 client.25670 shutdown 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 msgr2=0x5562fd6733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd6733f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5562fd6697f0 tx=0x7f782c009500 comp rx=0 tx=0).stop 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 shutdown_connections 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd27e1c0 0x5562fd684b50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd6733f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd672eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1072834673 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd672970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f7813400640 0 client.25670 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 >> 172.21.15.39:0/1072834673 conn(0x5562fd2b3990 msgr2=0x5562fd6639a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 shutdown_connections 2024-09-21T01:06:42.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.366+0000 7f783f5f8740 1 -- 172.21.15.39:0/1072834673 wait complete. 2024-09-21T01:06:42.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.367+0000 7f783f5f8740 20 client.25670 trim_cache size 0 max 16384 2024-09-21T01:06:42.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.385+0000 7f783f5f8740 1 Processor -- start 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.385+0000 7f783f5f8740 1 -- start start 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd674850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd6649b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd664ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f783f5f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f78301c25c0 con 0x5562fd29eb60 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f783f5f8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f78301bfe90 con 0x5562fd2bf840 2024-09-21T01:06:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f783f5f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f78301bdce0 con 0x5562fd296980 2024-09-21T01:06:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f7836800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd674850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f7836800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd674850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48496/0 (socket says 172.21.15.39:48496) 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f7836800640 1 -- 172.21.15.39:0/2109757868 learned_addr learned my addr 172.21.15.39:0/2109757868 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f7835400640 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd6649b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.386+0000 7f7836800640 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd664ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f7836800640 1 -- 172.21.15.39:0/2109757868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 msgr2=0x5562fd664ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f7836800640 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd664ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f7836800640 1 -- 172.21.15.39:0/2109757868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 msgr2=0x5562fd6649b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f7836800640 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd6649b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f7836800640 1 -- 172.21.15.39:0/2109757868 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562fd660c50 con 0x5562fd296980 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f7836800640 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd674850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f782c0118b0 tx=0x7f782c011d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f58fbeab8a22ae18 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.387+0000 7f781e600640 1 -- 172.21.15.39:0/2109757868 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c01e030 con 0x5562fd296980 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.388+0000 7f781e600640 1 -- 172.21.15.39:0/2109757868 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f782c010760 con 0x5562fd296980 2024-09-21T01:06:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.388+0000 7f781e600640 1 -- 172.21.15.39:0/2109757868 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c019b50 con 0x5562fd296980 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.388+0000 7f783f5f8740 1 -- 172.21.15.39:0/2109757868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 msgr2=0x5562fd674850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.388+0000 7f783f5f8740 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd674850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f782c0118b0 tx=0x7f782c011d70 comp rx=0 tx=0).stop 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 -- 172.21.15.39:0/2109757868 shutdown_connections 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd664ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd29eb60 0x5562fd6649b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 --2- 172.21.15.39:0/2109757868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd296980 0x5562fd674850 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 -- 172.21.15.39:0/2109757868 >> 172.21.15.39:0/2109757868 conn(0x5562fd2b3990 msgr2=0x5562fd67dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 -- 172.21.15.39:0/2109757868 shutdown_connections 2024-09-21T01:06:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 -- 172.21.15.39:0/2109757868 wait complete. 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.389+0000 7f783f5f8740 1 -- start start 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 Processor -- start 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd676e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd29eb60 0x5562fd6773d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd677910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f78301be350 con 0x5562fd296980 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f78301bf7c0 con 0x5562fd2bf840 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f78301bea20 con 0x5562fd29eb60 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f783f5f8740 10 client.0 _ll_register_callbacks cb 0x7f782c004d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.390+0000 7f7836800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd676e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd676e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35472/0 (socket says 172.21.15.39:35472) 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 -- 172.21.15.39:0/1174518627 learned_addr learned my addr 172.21.15.39:0/1174518627 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:06:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd677910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 -- 172.21.15.39:0/1174518627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd29eb60 msgr2=0x5562fd6773d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:06:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd29eb60 0x5562fd6773d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 -- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 msgr2=0x5562fd677910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd677910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 -- 172.21.15.39:0/1174518627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562fd665e00 con 0x5562fd296980 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.391+0000 7f7836800640 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd676e90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f782c009690 tx=0x7f782c009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f4b1854d95be10 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f7813400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f7813400640 1 -- 172.21.15.39:0/1174518627 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c01e040 con 0x5562fd296980 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f7813400640 1 -- 172.21.15.39:0/1174518627 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f782c010e10 con 0x5562fd296980 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f782c010760 con 0x5562fd296980 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f7813400640 1 -- 172.21.15.39:0/1174518627 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f782c00fb60 con 0x5562fd296980 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f783f5f8740 10 client.15645 fetch_fsmap learned FSMap version 28 2024-09-21T01:06:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.392+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562fd6ef4c0 con 0x5562fd296980 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f7813400640 1 -- 172.21.15.39:0/1174518627 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f782c010760 con 0x5562fd296980 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f7813400640 1 -- 172.21.15.39:0/1174518627 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f782c036040 con 0x5562fd296980 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f783f5f8740 10 client.15645 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f783f5f8740 10 client.15645 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f783f5f8740 20 client.15645 populate_metadata read hostname 'smithi039' 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd253570 0x5562fd6f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f783f5f8740 4 client.15645 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}] 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}) -- 0x5562fd5da6a0 con 0x5562fd253570 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.393+0000 7f7835400640 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd253570 0x5562fd6f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:06:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.394+0000 7f7813400640 10 client.15645 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:42.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.394+0000 7f7835400640 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd253570 0x5562fd6f1c60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=840bdf65bec1d6d4 server_cookie=b25f8c76748201e9 in_seq=0 out_seq=0 2024-09-21T01:06:42.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f7813400640 1 -- 172.21.15.39:0/1174518627 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5562fd5da6a0 con 0x5562fd253570 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f7813400640 10 client.15645 handle_command_reply: tid=0 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 client.15645 shutdown 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 msgr2=0x5562fd676e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd676e90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f782c009690 tx=0x7f782c009fa0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 shutdown_connections 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5562fd253570 0x5562fd6f1c60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562fd2bf840 0x5562fd677910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f7813400640 0 client.15645 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5562fd29eb60 0x5562fd6773d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 --2- 172.21.15.39:0/1174518627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562fd296980 0x5562fd676e90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 >> 172.21.15.39:0/1174518627 conn(0x5562fd2b3990 msgr2=0x5562fd674480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 shutdown_connections 2024-09-21T01:06:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.451+0000 7f783f5f8740 1 -- 172.21.15.39:0/1174518627 wait complete. 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f783f5f8740 20 client.15645 trim_cache size 0 max 16384 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 28, 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 14, 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "ccea113f": { 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 14, 2024-09-21T01:06:42.453 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 147.8, 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:42.455 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:06:42.455 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:06:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78180b6700 msgr2=0x7f78180b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78180b6700 0x7f78180b8c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5562fd665fd0 tx=0x7f7824000bc0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7830105d80 msgr2=0x7f783007b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7830105d80 0x7f783007b640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f782402a910 tx=0x7f7824031a20 comp rx=0 tx=0).stop 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 shutdown_connections 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f783010c010 0x7f7830078040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f783010b6c0 0x7f7830077920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78180b6700 0x7f78180b8c10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 --2- 172.21.15.39:0/2493774547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7830105d80 0x7f783007b640 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.452+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 >> 172.21.15.39:0/2493774547 conn(0x7f78301017d0 msgr2=0x7f7830103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.453+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 shutdown_connections 2024-09-21T01:06:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:06:42.453+0000 7f7838800640 1 -- 172.21.15.39:0/2493774547 wait complete. 2024-09-21T01:06:43.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 7.48 MB/sec warmup 70 sec latency 663.443 ms 2024-09-21T01:06:43.652 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ccea113f', quiesce took 5.0 seconds. Will release after: 22.8 2024-09-21T01:06:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:43 smithi150 ceph-mon[34945]: pgmap v485: 129 pgs: 129 active+clean; 64 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 111 op/s 2024-09-21T01:06:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:43 smithi160 ceph-mon[34696]: pgmap v485: 129 pgs: 129 active+clean; 64 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 111 op/s 2024-09-21T01:06:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:43 smithi039 ceph-mon[30415]: pgmap v485: 129 pgs: 129 active+clean; 64 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 111 op/s 2024-09-21T01:06:44.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 7.37 MB/sec warmup 71 sec latency 1663.566 ms 2024-09-21T01:06:45.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 7.27 MB/sec warmup 72 sec latency 2663.760 ms 2024-09-21T01:06:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:45 smithi150 ceph-mon[34945]: pgmap v486: 129 pgs: 129 active+clean; 62 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 112 op/s 2024-09-21T01:06:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:45 smithi160 ceph-mon[34696]: pgmap v486: 129 pgs: 129 active+clean; 62 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 112 op/s 2024-09-21T01:06:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:45 smithi039 ceph-mon[30415]: pgmap v486: 129 pgs: 129 active+clean; 62 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 112 op/s 2024-09-21T01:06:46.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 7.17 MB/sec warmup 73 sec latency 3663.834 ms 2024-09-21T01:06:47.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 7.08 MB/sec warmup 74 sec latency 4663.914 ms 2024-09-21T01:06:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:47 smithi150 ceph-mon[34945]: pgmap v487: 129 pgs: 129 active+clean; 62 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-09-21T01:06:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:47 smithi160 ceph-mon[34696]: pgmap v487: 129 pgs: 129 active+clean; 62 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-09-21T01:06:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:47 smithi039 ceph-mon[30415]: pgmap v487: 129 pgs: 129 active+clean; 62 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-09-21T01:06:48.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.98 MB/sec warmup 75 sec latency 5664.023 ms 2024-09-21T01:06:49.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.89 MB/sec warmup 76 sec latency 6664.165 ms 2024-09-21T01:06:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:49 smithi150 ceph-mon[34945]: pgmap v488: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:06:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:49 smithi160 ceph-mon[34696]: pgmap v488: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:06:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:49 smithi039 ceph-mon[30415]: pgmap v488: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:06:50.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.80 MB/sec warmup 77 sec latency 7664.312 ms 2024-09-21T01:06:51.106 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.71 MB/sec warmup 78 sec latency 8664.414 ms 2024-09-21T01:06:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:51 smithi150 ceph-mon[34945]: pgmap v489: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-09-21T01:06:51.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:51 smithi160 ceph-mon[34696]: pgmap v489: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-09-21T01:06:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:51 smithi039 ceph-mon[30415]: pgmap v489: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-09-21T01:06:52.107 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.63 MB/sec warmup 79 sec latency 9664.542 ms 2024-09-21T01:06:53.107 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.54 MB/sec warmup 80 sec latency 10664.699 ms 2024-09-21T01:06:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:53 smithi150 ceph-mon[34945]: pgmap v490: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-09-21T01:06:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:53 smithi160 ceph-mon[34696]: pgmap v490: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-09-21T01:06:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:53 smithi039 ceph-mon[30415]: pgmap v490: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-09-21T01:06:54.107 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.46 MB/sec warmup 81 sec latency 11664.791 ms 2024-09-21T01:06:55.107 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.39 MB/sec warmup 82 sec latency 12664.932 ms 2024-09-21T01:06:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:55 smithi150 ceph-mon[34945]: pgmap v491: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 500 KiB/s wr, 55 op/s 2024-09-21T01:06:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:55 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:06:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:55 smithi160 ceph-mon[34696]: pgmap v491: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 500 KiB/s wr, 55 op/s 2024-09-21T01:06:55.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:55 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:06:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:55 smithi039 ceph-mon[30415]: pgmap v491: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 500 KiB/s wr, 55 op/s 2024-09-21T01:06:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:55 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:06:56.107 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.31 MB/sec warmup 83 sec latency 13664.997 ms 2024-09-21T01:06:56.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:06:56.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:06:56.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:06:56.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:06:56.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:06:56.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:06:56.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:06:56.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:06:56.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:06:57.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.23 MB/sec warmup 84 sec latency 14665.126 ms 2024-09-21T01:06:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:57 smithi150 ceph-mon[34945]: pgmap v492: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 324 KiB/s wr, 41 op/s 2024-09-21T01:06:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:57 smithi160 ceph-mon[34696]: pgmap v492: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 324 KiB/s wr, 41 op/s 2024-09-21T01:06:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:57 smithi039 ceph-mon[30415]: pgmap v492: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 324 KiB/s wr, 41 op/s 2024-09-21T01:06:58.107 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.16 MB/sec warmup 85 sec latency 15665.270 ms 2024-09-21T01:06:59.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.09 MB/sec warmup 86 sec latency 16665.399 ms 2024-09-21T01:06:59.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:06:59 smithi150 ceph-mon[34945]: pgmap v493: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 100 KiB/s wr, 24 op/s 2024-09-21T01:06:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:06:59 smithi160 ceph-mon[34696]: pgmap v493: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 100 KiB/s wr, 24 op/s 2024-09-21T01:06:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:06:59 smithi039 ceph-mon[30415]: pgmap v493: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 100 KiB/s wr, 24 op/s 2024-09-21T01:07:00.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 6.02 MB/sec warmup 87 sec latency 17665.523 ms 2024-09-21T01:07:01.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.95 MB/sec warmup 88 sec latency 18665.666 ms 2024-09-21T01:07:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:01 smithi150 ceph-mon[34945]: pgmap v494: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-09-21T01:07:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:01 smithi160 ceph-mon[34696]: pgmap v494: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-09-21T01:07:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:01 smithi039 ceph-mon[30415]: pgmap v494: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-09-21T01:07:02.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.88 MB/sec warmup 89 sec latency 19665.836 ms 2024-09-21T01:07:03.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.82 MB/sec warmup 90 sec latency 20665.924 ms 2024-09-21T01:07:03.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:03 smithi039 ceph-mon[30415]: pgmap v495: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 KiB/s wr, 10 op/s 2024-09-21T01:07:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:03 smithi150 ceph-mon[34945]: pgmap v495: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 KiB/s wr, 10 op/s 2024-09-21T01:07:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:03 smithi160 ceph-mon[34696]: pgmap v495: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 KiB/s wr, 10 op/s 2024-09-21T01:07:04.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.75 MB/sec warmup 91 sec latency 21665.964 ms 2024-09-21T01:07:05.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.69 MB/sec warmup 92 sec latency 22666.106 ms 2024-09-21T01:07:05.746 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:05 smithi039 ceph-mon[30415]: pgmap v496: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-09-21T01:07:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:05 smithi150 ceph-mon[34945]: pgmap v496: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-09-21T01:07:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:05 smithi160 ceph-mon[34696]: pgmap v496: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-09-21T01:07:06.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.63 MB/sec warmup 93 sec latency 23666.171 ms 2024-09-21T01:07:06.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:07:06.841 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:07:07.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.57 MB/sec warmup 94 sec latency 24666.209 ms 2024-09-21T01:07:07.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 -- 172.21.15.39:0/2453916712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb641094a0 msgr2=0x7feb641098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:07.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 --2- 172.21.15.39:0/2453916712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb641094a0 0x7feb641098b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009900 tx=0x7feb4c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 -- 172.21.15.39:0/2453916712 shutdown_connections 2024-09-21T01:07:07.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 --2- 172.21.15.39:0/2453916712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64109df0 0x7feb64110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 --2- 172.21.15.39:0/2453916712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb641094a0 0x7feb641098b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 --2- 172.21.15.39:0/2453916712 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb64103b60 0x7feb64103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 -- 172.21.15.39:0/2453916712 >> 172.21.15.39:0/2453916712 conn(0x7feb6407b850 msgr2=0x7feb6407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:07.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.488+0000 7feb6aa00640 1 -- 172.21.15.39:0/2453916712 shutdown_connections 2024-09-21T01:07:07.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.489+0000 7feb6aa00640 1 -- 172.21.15.39:0/2453916712 wait complete. 2024-09-21T01:07:07.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 Processor -- start 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 -- start start 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 0x7feb64075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb641094a0 0x7feb64075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb64109df0 0x7feb64076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7feb64113ab0 con 0x7feb64109df0 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb64113930 con 0x7feb64103b60 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.490+0000 7feb6aa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feb64113c30 con 0x7feb641094a0 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 0x7feb64075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb68a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb64109df0 0x7feb64076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 0x7feb64075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:58214/0 (socket says 172.21.15.39:58214) 2024-09-21T01:07:07.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 -- 172.21.15.39:0/4066243050 learned_addr learned my addr 172.21.15.39:0/4066243050 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb5be00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb641094a0 0x7feb64075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 -- 172.21.15.39:0/4066243050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb641094a0 msgr2=0x7feb64075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb641094a0 0x7feb64075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 -- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb64109df0 msgr2=0x7feb64076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb64109df0 0x7feb64076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb63e00640 1 -- 172.21.15.39:0/4066243050 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb641a8760 con 0x7feb64103b60 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.491+0000 7feb68a00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb64109df0 0x7feb64076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:07:07.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.492+0000 7feb63e00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 0x7feb64075710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb5400e9f0 tx=0x7feb5400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.492+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5400cd80 con 0x7feb64103b60 2024-09-21T01:07:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.492+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb540054d0 con 0x7feb64103b60 2024-09-21T01:07:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.492+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb641a8a50 con 0x7feb64103b60 2024-09-21T01:07:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.492+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54010690 con 0x7feb64103b60 2024-09-21T01:07:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.492+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb641a8f60 con 0x7feb64103b60 2024-09-21T01:07:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.493+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb64103f70 con 0x7feb64103b60 2024-09-21T01:07:07.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.496+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7feb54004e30 con 0x7feb64103b60 2024-09-21T01:07:07.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.497+0000 7feb61600640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feb480b6700 0x7feb480b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:07.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.498+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7feb540e7ff0 con 0x7feb64103b60 2024-09-21T01:07:07.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.498+0000 7feb5be00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feb480b6700 0x7feb480b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:07.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.498+0000 7feb5be00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feb480b6700 0x7feb480b8c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009230 tx=0x7feb4c031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:07.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.500+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7feb540ae1a0 con 0x7feb64103b60 2024-09-21T01:07:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:07 smithi039 ceph-mon[30415]: pgmap v497: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-09-21T01:07:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.755+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb64076a10 con 0x7feb64103b60 2024-09-21T01:07:07.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.756+0000 7feb61600640 1 -- 172.21.15.39:0/4066243050 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11914 (secure 0 0 0) 0x7feb64076a10 con 0x7feb64103b60 2024-09-21T01:07:07.758 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:07:07.758 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"btime":"2024-09-21T01:05:55:033931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:05:54.497271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:07:07.758 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feb480b6700 msgr2=0x7feb480b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feb480b6700 0x7feb480b8c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009230 tx=0x7feb4c031400 comp rx=0 tx=0).stop 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 msgr2=0x7feb64075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 0x7feb64075710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb5400e9f0 tx=0x7feb5400edf0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 shutdown_connections 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb64109df0 0x7feb64076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb641094a0 0x7feb64075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feb480b6700 0x7feb480b8c10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 --2- 172.21.15.39:0/4066243050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb64103b60 0x7feb64075710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 >> 172.21.15.39:0/4066243050 conn(0x7feb6407b850 msgr2=0x7feb64107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 shutdown_connections 2024-09-21T01:07:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:07.759+0000 7feb6aa00640 1 -- 172.21.15.39:0/4066243050 wait complete. 2024-09-21T01:07:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:07 smithi150 ceph-mon[34945]: pgmap v497: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-09-21T01:07:07.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:07 smithi160 ceph-mon[34696]: pgmap v497: 129 pgs: 129 active+clean; 59 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-09-21T01:07:08.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.51 MB/sec warmup 95 sec latency 25666.314 ms 2024-09-21T01:07:08.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:08 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4066243050' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:07:08.848 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:08 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4066243050' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:07:08.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:08 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4066243050' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:07:08.891 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id ccea113f --release --await' 2024-09-21T01:07:08.891 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id ccea113f --release --await 2024-09-21T01:07:09.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.45 MB/sec warmup 96 sec latency 26666.377 ms 2024-09-21T01:07:09.275 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:07:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:09 smithi150 ceph-mon[34945]: pgmap v498: 129 pgs: 129 active+clean; 62 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 326 KiB/s wr, 27 op/s 2024-09-21T01:07:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:09 smithi160 ceph-mon[34696]: pgmap v498: 129 pgs: 129 active+clean; 62 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 326 KiB/s wr, 27 op/s 2024-09-21T01:07:09.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:09 smithi039 ceph-mon[30415]: pgmap v498: 129 pgs: 129 active+clean; 62 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 326 KiB/s wr, 27 op/s 2024-09-21T01:07:10.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 -- 172.21.15.39:0/2980224672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c41092d0 msgr2=0x7fd9c41096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/2980224672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c41092d0 0x7fd9c41096e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009900 tx=0x7fd9b002a410 comp rx=0 tx=0).stop 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 -- 172.21.15.39:0/2980224672 shutdown_connections 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/2980224672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9c4109c20 0x7fd9c4110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/2980224672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c41092d0 0x7fd9c41096e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/2980224672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9c4103990 0x7fd9c4103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 -- 172.21.15.39:0/2980224672 >> 172.21.15.39:0/2980224672 conn(0x7fd9c40ff3e0 msgr2=0x7fd9c4101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 -- 172.21.15.39:0/2980224672 shutdown_connections 2024-09-21T01:07:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.073+0000 7fd9c9c00640 1 -- 172.21.15.39:0/2980224672 wait complete. 2024-09-21T01:07:10.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.074+0000 7fd9c9c00640 1 Processor -- start 2024-09-21T01:07:10.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.074+0000 7fd9c9c00640 1 -- start start 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 0x7fd9c419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9c41092d0 0x7fd9c419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9c4109c20 0x7fd9c419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41138e0 con 0x7fd9c4103990 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c4113760 con 0x7fd9c4109c20 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c4113a60 con 0x7fd9c41092d0 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 0x7fd9c419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 0x7fd9c419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46804/0 (socket says 172.21.15.39:46804) 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3125242931 learned_addr learned my addr 172.21.15.39:0/3125242931 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9c41092d0 0x7fd9c419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c9200640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9c4109c20 0x7fd9c419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3125242931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9c41092d0 msgr2=0x7fd9c419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9c41092d0 0x7fd9c419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3125242931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9c4109c20 msgr2=0x7fd9c419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9c4109c20 0x7fd9c419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.075+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3125242931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c41a0280 con 0x7fd9c4103990 2024-09-21T01:07:10.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.076+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 0x7fd9c419a310 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400e9f0 tx=0x7fd9b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:10.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.076+0000 7fd9c0c00640 1 -- 172.21.15.39:0/3125242931 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b400cd80 con 0x7fd9c4103990 2024-09-21T01:07:10.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.076+0000 7fd9c0c00640 1 -- 172.21.15.39:0/3125242931 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9b40108e0 con 0x7fd9c4103990 2024-09-21T01:07:10.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.076+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c41a0570 con 0x7fd9c4103990 2024-09-21T01:07:10.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.076+0000 7fd9c0c00640 1 -- 172.21.15.39:0/3125242931 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4018980 con 0x7fd9c4103990 2024-09-21T01:07:10.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.076+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c41a6cd0 con 0x7fd9c4103990 2024-09-21T01:07:10.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.079+0000 7fd9d1e40740 1 Processor -- start 2024-09-21T01:07:10.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 -- start start 2024-09-21T01:07:10.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9c0c00640 1 -- 172.21.15.39:0/3125242931 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fd9b4010430 con 0x7fd9c4103990 2024-09-21T01:07:10.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc10340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 0x55e2fdc10880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd848840 0x55e2fdc10dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e2fdb87290 con 0x55e2fd840660 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e2fdb766c0 con 0x55e2fd848840 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.080+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e2fdb7a4d0 con 0x55e2fd869520 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 0x55e2fdc10880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c9200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc10340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 0x55e2fdc10880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46806/0 (socket says 172.21.15.39:46806) 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3188465882 learned_addr learned my addr 172.21.15.39:0/3188465882 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:07:10.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd848840 0x55e2fdc10dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3188465882 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 msgr2=0x55e2fdc10340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc10340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3188465882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd848840 msgr2=0x55e2fdc10dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd848840 0x55e2fdc10dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.081+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3188465882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2fdb03460 con 0x55e2fd840660 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 0x55e2fdc10880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40ebf10 tx=0x7fd9b40f04b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1287df24e41882 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9c0c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9a80b6680 0x7fd9a80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9b9600640 1 -- 172.21.15.39:0/3188465882 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40fb070 con 0x55e2fd840660 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9c0c00640 1 -- 172.21.15.39:0/3125242931 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9b40e7e30 con 0x7fd9c4103990 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9c9200640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9a80b6680 0x7fd9a80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9b9600640 1 -- 172.21.15.39:0/3188465882 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9b4010720 con 0x55e2fd840660 2024-09-21T01:07:10.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.082+0000 7fd9b9600640 1 -- 172.21.15.39:0/3188465882 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40a97f0 con 0x55e2fd840660 2024-09-21T01:07:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.083+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3188465882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 msgr2=0x55e2fdc10880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.083+0000 7fd9c9200640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9a80b6680 0x7fd9a80b8b90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc005fd0 tx=0x7fd9bc009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.083+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 0x55e2fdc10880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40ebf10 tx=0x7fd9b40f04b0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3188465882 shutdown_connections 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd848840 0x55e2fdc10dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd840660 0x55e2fdc10880 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3188465882 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc10340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3188465882 >> 172.21.15.39:0/3188465882 conn(0x55e2fd85d910 msgr2=0x55e2fdc0e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3188465882 shutdown_connections 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3188465882 wait complete. 2024-09-21T01:07:10.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 -- start start 2024-09-21T01:07:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.084+0000 7fd9d1e40740 1 Processor -- start 2024-09-21T01:07:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc202c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc20800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc2c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e2fd9d7bb0 con 0x55e2fd848840 2024-09-21T01:07:10.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e2fdb116c0 con 0x55e2fd869520 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e2fdc22e50 con 0x55e2fd840660 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9d1e40740 10 client.0 _ll_register_callbacks cb 0x55e2fdb2c490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc202c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc202c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:38386/0 (socket says 172.21.15.39:38386) 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3886225633 learned_addr learned my addr 172.21.15.39:0/3886225633 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9c3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc20800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.085+0000 7fd9c9200640 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc2c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 msgr2=0x55e2fdc2c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc2c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3886225633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 msgr2=0x55e2fdc20800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc20800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3886225633 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2fd8d4b00 con 0x55e2fd840660 2024-09-21T01:07:10.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc202c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400e9c0 tx=0x7fd9b4018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3cd8a6f27449d4d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:10.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9ae000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:07:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9ae000640 1 -- 172.21.15.39:0/3886225633 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40ad0c0 con 0x55e2fd840660 2024-09-21T01:07:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9ae000640 1 -- 172.21.15.39:0/3886225633 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9b40aebd0 con 0x55e2fd840660 2024-09-21T01:07:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.086+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2fdb891e0 con 0x55e2fd840660 2024-09-21T01:07:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.087+0000 7fd9ae000640 1 -- 172.21.15.39:0/3886225633 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9b40ad2a0 con 0x55e2fd840660 2024-09-21T01:07:10.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.087+0000 7fd9d1e40740 10 client.25681 fetch_fsmap learned FSMap version 28 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.087+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2fd94b960 con 0x55e2fd840660 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.087+0000 7fd9ae000640 1 -- 172.21.15.39:0/3886225633 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e2fdb891e0 con 0x55e2fd840660 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.087+0000 7fd9ae000640 1 -- 172.21.15.39:0/3886225633 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd9b40b6e30 con 0x55e2fd840660 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9d1e40740 10 client.25681 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9d1e40740 10 client.25681 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9d1e40740 20 client.25681 populate_metadata read hostname 'smithi039' 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd827ea0 0x55e2fdc32040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9d1e40740 4 client.25681 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2fdb84c90 con 0x55e2fd827ea0 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.088+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd827ea0 0x55e2fdc32040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.089+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd827ea0 0x55e2fdc32040 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1fff03bebedefb6 server_cookie=ae6b729ee6b93f10 in_seq=0 out_seq=0 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.089+0000 7fd9ae000640 10 client.25681 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.089+0000 7fd9ae000640 1 -- 172.21.15.39:0/3886225633 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e2fdb84c90 con 0x55e2fd827ea0 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.089+0000 7fd9ae000640 10 client.25681 handle_command_reply: tid=0 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.089+0000 7fd9d1e40740 1 client.25681 shutdown 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 msgr2=0x55e2fdc202c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc202c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400e9c0 tx=0x7fd9b4018420 comp rx=0 tx=0).stop 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 shutdown_connections 2024-09-21T01:07:10.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd827ea0 0x55e2fdc32040 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc2c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc20800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3886225633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc202c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9ae000640 0 client.25681 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 >> 172.21.15.39:0/3886225633 conn(0x55e2fd85d910 msgr2=0x55e2fdc13490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 shutdown_connections 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3886225633 wait complete. 2024-09-21T01:07:10.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.090+0000 7fd9d1e40740 20 client.25681 trim_cache size 0 max 16384 2024-09-21T01:07:10.108 INFO:tasks.workunit.client.0.smithi039.stdout: 1 94115 5.40 MB/sec warmup 97 sec latency 27666.441 ms 2024-09-21T01:07:10.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 Processor -- start 2024-09-21T01:07:10.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 -- start start 2024-09-21T01:07:10.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc21cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc1d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc1d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41b13f0 con 0x55e2fd848840 2024-09-21T01:07:10.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41aecc0 con 0x55e2fd869520 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41acb10 con 0x55e2fd840660 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9c3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc1d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9c3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc1d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46822/0 (socket says 172.21.15.39:46822) 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9c3400640 1 -- 172.21.15.39:0/1075286672 learned_addr learned my addr 172.21.15.39:0/1075286672 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.109+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc21cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c9200640 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc1d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3e00640 1 -- 172.21.15.39:0/1075286672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 msgr2=0x55e2fdc1d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc1d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3e00640 1 -- 172.21.15.39:0/1075286672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 msgr2=0x55e2fdc1d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc1d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3e00640 1 -- 172.21.15.39:0/1075286672 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2fdc14380 con 0x55e2fd840660 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3400640 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc1d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:07:10.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.110+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc21cf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400c6a0 tx=0x7fd9b40f04b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c28651d9f6b98f0a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:10.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.111+0000 7fd9b8c00640 1 -- 172.21.15.39:0/1075286672 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40ad720 con 0x55e2fd840660 2024-09-21T01:07:10.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.111+0000 7fd9b8c00640 1 -- 172.21.15.39:0/1075286672 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9b40eb410 con 0x55e2fd840660 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.111+0000 7fd9b8c00640 1 -- 172.21.15.39:0/1075286672 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40a97f0 con 0x55e2fd840660 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.111+0000 7fd9d1e40740 1 -- 172.21.15.39:0/1075286672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 msgr2=0x55e2fdc21cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.111+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc21cf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400c6a0 tx=0x7fd9b40f04b0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 -- 172.21.15.39:0/1075286672 shutdown_connections 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd869520 0x55e2fdc1d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc1d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/1075286672 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd840660 0x55e2fdc21cf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 -- 172.21.15.39:0/1075286672 >> 172.21.15.39:0/1075286672 conn(0x55e2fd85d910 msgr2=0x55e2fdc215e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 -- 172.21.15.39:0/1075286672 shutdown_connections 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 -- 172.21.15.39:0/1075286672 wait complete. 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 -- start start 2024-09-21T01:07:10.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.112+0000 7fd9d1e40740 1 Processor -- start 2024-09-21T01:07:10.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 0x55e2fdc34740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc34c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc9c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41ad180 con 0x55e2fd848840 2024-09-21T01:07:10.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41ae5f0 con 0x55e2fd840660 2024-09-21T01:07:10.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41ad850 con 0x55e2fd869520 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9d1e40740 10 client.0 _ll_register_callbacks cb 0x7fd9b40f9070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 0x55e2fdc34740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9c3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 0x55e2fdc34740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:58270/0 (socket says 172.21.15.39:58270) 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3730983062 learned_addr learned my addr 172.21.15.39:0/3730983062 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc34c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.113+0000 7fd9c9200640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc9c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3730983062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 msgr2=0x55e2fdc9c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc9c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3730983062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 msgr2=0x55e2fdc34c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc34c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9c3e00640 1 -- 172.21.15.39:0/3730983062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2fdc9cd10 con 0x55e2fd840660 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9c3e00640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 0x55e2fdc34740 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40aabc0 tx=0x7fd9b4018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9479a8e52c9a6509 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9ae000640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:07:10.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9ae000640 1 -- 172.21.15.39:0/3730983062 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40a9fa0 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2fdb820c0 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9ae000640 1 -- 172.21.15.39:0/3730983062 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9b40aedd0 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9ae000640 1 -- 172.21.15.39:0/3730983062 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9b40a90c0 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9d1e40740 10 client.25700 fetch_fsmap learned FSMap version 28 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.114+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2fdc9cd10 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9ae000640 1 -- 172.21.15.39:0/3730983062 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e2fdb820c0 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9ae000640 1 -- 172.21.15.39:0/3730983062 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd9b4018b20 con 0x55e2fd840660 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9d1e40740 10 client.25700 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9d1e40740 10 client.25700 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9d1e40740 20 client.25700 populate_metadata read hostname 'smithi039' 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd7fd250 0x55e2fdca02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9d1e40740 4 client.25700 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ccea113f", "release": true, "await": true}] 2024-09-21T01:07:10.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ccea113f", "release": true, "await": true}) -- 0x55e2fdaf7550 con 0x55e2fd7fd250 2024-09-21T01:07:10.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.115+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd7fd250 0x55e2fdca02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:07:10.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.116+0000 7fd9c3400640 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd7fd250 0x55e2fdca02b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328d3d58e338fde1 server_cookie=a0698f4762d77141 in_seq=0 out_seq=0 2024-09-21T01:07:10.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.116+0000 7fd9ae000640 10 client.25700 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:07:10.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9ae000640 1 -- 172.21.15.39:0/3730983062 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e2fdaf7550 con 0x55e2fd7fd250 2024-09-21T01:07:10.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9ae000640 10 client.25700 handle_command_reply: tid=0 2024-09-21T01:07:10.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 client.25700 shutdown 2024-09-21T01:07:10.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 msgr2=0x55e2fdc34740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 0x55e2fdc34740 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40aabc0 tx=0x7fd9b4018420 comp rx=0 tx=0).stop 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 shutdown_connections 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e2fd7fd250 0x55e2fdca02b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e2fd869520 0x55e2fdc9c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e2fd848840 0x55e2fdc34c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9ae000640 0 client.25700 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 --2- 172.21.15.39:0/3730983062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2fd840660 0x55e2fdc34740 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.130+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 >> 172.21.15.39:0/3730983062 conn(0x55e2fd85d910 msgr2=0x55e2fdc11350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.131+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 shutdown_connections 2024-09-21T01:07:10.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.131+0000 7fd9d1e40740 1 -- 172.21.15.39:0/3730983062 wait complete. 2024-09-21T01:07:10.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.131+0000 7fd9d1e40740 20 client.25700 trim_cache size 0 max 16384 2024-09-21T01:07:10.134 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:07:10.134 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 28, 2024-09-21T01:07:10.134 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 16, 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "ccea113f": { 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 16, 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:07:10.135 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 147.8, 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:07:10.136 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:07:10.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.131+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9a80b6680 msgr2=0x7fd9a80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.131+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9a80b6680 0x7fd9a80b8b90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc005fd0 tx=0x7fd9bc009500 comp rx=0 tx=0).stop 2024-09-21T01:07:10.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 msgr2=0x7fd9c419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:07:10.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 0x7fd9c419a310 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400e9f0 tx=0x7fd9b400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 shutdown_connections 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd9a80b6680 0x7fd9a80b8b90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9c4109c20 0x7fd9c419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd9c41092d0 0x7fd9c419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 --2- 172.21.15.39:0/3125242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9c4103990 0x7fd9c419a310 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 >> 172.21.15.39:0/3125242931 conn(0x7fd9c40ff3e0 msgr2=0x7fd9c4101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 shutdown_connections 2024-09-21T01:07:10.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:07:10.132+0000 7fd9c9c00640 1 -- 172.21.15.39:0/3125242931 wait complete. 2024-09-21T01:07:10.989 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ccea113f', total seconds elapsed: 32.3 2024-09-21T01:07:11.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 96126 5.48 MB/sec warmup 98 sec latency 27682.729 ms 2024-09-21T01:07:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:11 smithi150 ceph-mon[34945]: pgmap v499: 129 pgs: 129 active+clean; 62 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 402 KiB/s wr, 29 op/s 2024-09-21T01:07:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:11 smithi160 ceph-mon[34696]: pgmap v499: 129 pgs: 129 active+clean; 62 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 402 KiB/s wr, 29 op/s 2024-09-21T01:07:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:11 smithi039 ceph-mon[30415]: pgmap v499: 129 pgs: 129 active+clean; 62 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 402 KiB/s wr, 29 op/s 2024-09-21T01:07:12.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 98109 5.50 MB/sec warmup 99 sec latency 15.421 ms 2024-09-21T01:07:13.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 99981 5.56 MB/sec warmup 100 sec latency 10.513 ms 2024-09-21T01:07:13.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:13 smithi150 ceph-mon[34945]: pgmap v500: 129 pgs: 129 active+clean; 61 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 519 KiB/s wr, 43 op/s 2024-09-21T01:07:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:13 smithi160 ceph-mon[34696]: pgmap v500: 129 pgs: 129 active+clean; 61 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 519 KiB/s wr, 43 op/s 2024-09-21T01:07:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:13 smithi039 ceph-mon[30415]: pgmap v500: 129 pgs: 129 active+clean; 61 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 519 KiB/s wr, 43 op/s 2024-09-21T01:07:14.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 102237 5.62 MB/sec warmup 101 sec latency 15.251 ms 2024-09-21T01:07:15.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 103783 5.65 MB/sec warmup 102 sec latency 14.901 ms 2024-09-21T01:07:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:15 smithi150 ceph-mon[34945]: pgmap v501: 129 pgs: 129 active+clean; 60 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 580 KiB/s wr, 48 op/s 2024-09-21T01:07:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:15 smithi160 ceph-mon[34696]: pgmap v501: 129 pgs: 129 active+clean; 60 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 580 KiB/s wr, 48 op/s 2024-09-21T01:07:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:15 smithi039 ceph-mon[30415]: pgmap v501: 129 pgs: 129 active+clean; 60 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 580 KiB/s wr, 48 op/s 2024-09-21T01:07:16.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 106435 5.76 MB/sec warmup 103 sec latency 12.274 ms 2024-09-21T01:07:17.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 108021 5.77 MB/sec warmup 104 sec latency 14.933 ms 2024-09-21T01:07:17.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:17 smithi150 ceph-mon[34945]: pgmap v502: 129 pgs: 129 active+clean; 62 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 858 KiB/s wr, 75 op/s 2024-09-21T01:07:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:17 smithi160 ceph-mon[34696]: pgmap v502: 129 pgs: 129 active+clean; 62 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 858 KiB/s wr, 75 op/s 2024-09-21T01:07:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:17 smithi039 ceph-mon[30415]: pgmap v502: 129 pgs: 129 active+clean; 62 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 858 KiB/s wr, 75 op/s 2024-09-21T01:07:18.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 110380 5.85 MB/sec warmup 105 sec latency 10.399 ms 2024-09-21T01:07:19.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 112397 5.86 MB/sec warmup 106 sec latency 26.036 ms 2024-09-21T01:07:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:19 smithi150 ceph-mon[34945]: pgmap v503: 129 pgs: 129 active+clean; 65 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-09-21T01:07:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:19 smithi160 ceph-mon[34696]: pgmap v503: 129 pgs: 129 active+clean; 65 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-09-21T01:07:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:19 smithi039 ceph-mon[30415]: pgmap v503: 129 pgs: 129 active+clean; 65 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-09-21T01:07:20.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 114210 5.92 MB/sec warmup 107 sec latency 18.722 ms 2024-09-21T01:07:21.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 116471 5.98 MB/sec warmup 108 sec latency 19.229 ms 2024-09-21T01:07:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:21 smithi150 ceph-mon[34945]: pgmap v504: 129 pgs: 129 active+clean; 68 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-09-21T01:07:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:21 smithi160 ceph-mon[34696]: pgmap v504: 129 pgs: 129 active+clean; 68 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-09-21T01:07:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:21 smithi039 ceph-mon[30415]: pgmap v504: 129 pgs: 129 active+clean; 68 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-09-21T01:07:22.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 118034 5.99 MB/sec warmup 109 sec latency 21.821 ms 2024-09-21T01:07:23.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 120580 6.09 MB/sec warmup 110 sec latency 31.493 ms 2024-09-21T01:07:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:23 smithi150 ceph-mon[34945]: pgmap v505: 129 pgs: 129 active+clean; 70 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 171 op/s 2024-09-21T01:07:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:23 smithi160 ceph-mon[34696]: pgmap v505: 129 pgs: 129 active+clean; 70 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 171 op/s 2024-09-21T01:07:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:23 smithi039 ceph-mon[30415]: pgmap v505: 129 pgs: 129 active+clean; 70 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.2 MiB/s wr, 171 op/s 2024-09-21T01:07:24.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 122126 6.09 MB/sec warmup 111 sec latency 16.894 ms 2024-09-21T01:07:25.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 124406 6.16 MB/sec warmup 112 sec latency 32.888 ms 2024-09-21T01:07:25.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:25 smithi150 ceph-mon[34945]: pgmap v506: 129 pgs: 129 active+clean; 74 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 218 op/s 2024-09-21T01:07:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:25 smithi160 ceph-mon[34696]: pgmap v506: 129 pgs: 129 active+clean; 74 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 218 op/s 2024-09-21T01:07:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:25 smithi039 ceph-mon[30415]: pgmap v506: 129 pgs: 129 active+clean; 74 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 218 op/s 2024-09-21T01:07:26.109 INFO:tasks.workunit.client.0.smithi039.stdout: 1 125997 6.16 MB/sec warmup 113 sec latency 14.989 ms 2024-09-21T01:07:27.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 128212 6.22 MB/sec warmup 114 sec latency 10.373 ms 2024-09-21T01:07:27.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:27 smithi150 ceph-mon[34945]: pgmap v507: 129 pgs: 129 active+clean; 74 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 217 op/s 2024-09-21T01:07:27.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:27 smithi160 ceph-mon[34696]: pgmap v507: 129 pgs: 129 active+clean; 74 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 217 op/s 2024-09-21T01:07:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:27 smithi039 ceph-mon[30415]: pgmap v507: 129 pgs: 129 active+clean; 74 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 217 op/s 2024-09-21T01:07:28.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 130506 6.26 MB/sec warmup 115 sec latency 14.163 ms 2024-09-21T01:07:29.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 132151 6.29 MB/sec warmup 116 sec latency 19.806 ms 2024-09-21T01:07:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:29 smithi150 ceph-mon[34945]: pgmap v508: 129 pgs: 129 active+clean; 77 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 218 op/s 2024-09-21T01:07:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:29 smithi160 ceph-mon[34696]: pgmap v508: 129 pgs: 129 active+clean; 77 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 218 op/s 2024-09-21T01:07:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:29 smithi039 ceph-mon[30415]: pgmap v508: 129 pgs: 129 active+clean; 77 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 218 op/s 2024-09-21T01:07:30.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 134647 6.35 MB/sec warmup 117 sec latency 17.245 ms 2024-09-21T01:07:31.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 136212 6.38 MB/sec warmup 118 sec latency 31.396 ms 2024-09-21T01:07:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:31 smithi150 ceph-mon[34945]: pgmap v509: 129 pgs: 129 active+clean; 82 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 231 op/s 2024-09-21T01:07:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:31 smithi160 ceph-mon[34696]: pgmap v509: 129 pgs: 129 active+clean; 82 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 231 op/s 2024-09-21T01:07:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:31 smithi039 ceph-mon[30415]: pgmap v509: 129 pgs: 129 active+clean; 82 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 231 op/s 2024-09-21T01:07:32.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 138637 6.44 MB/sec warmup 119 sec latency 10.923 ms 2024-09-21T01:07:33.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:33 smithi039 ceph-mon[30415]: pgmap v510: 129 pgs: 129 active+clean; 83 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-09-21T01:07:33.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:33 smithi150 ceph-mon[34945]: pgmap v510: 129 pgs: 129 active+clean; 83 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-09-21T01:07:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:33 smithi160 ceph-mon[34696]: pgmap v510: 129 pgs: 129 active+clean; 83 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-09-21T01:07:34.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 142504 13.64 MB/sec execute 1 sec latency 15.158 ms 2024-09-21T01:07:35.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 144838 13.08 MB/sec execute 2 sec latency 15.545 ms 2024-09-21T01:07:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:35 smithi150 ceph-mon[34945]: pgmap v511: 129 pgs: 129 active+clean; 85 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-09-21T01:07:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:35 smithi160 ceph-mon[34696]: pgmap v511: 129 pgs: 129 active+clean; 85 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-09-21T01:07:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:35 smithi039 ceph-mon[30415]: pgmap v511: 129 pgs: 129 active+clean; 85 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-09-21T01:07:36.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 146346 11.15 MB/sec execute 3 sec latency 34.891 ms 2024-09-21T01:07:37.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 148948 12.53 MB/sec execute 4 sec latency 12.783 ms 2024-09-21T01:07:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:37 smithi150 ceph-mon[34945]: pgmap v512: 129 pgs: 129 active+clean; 88 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 199 op/s 2024-09-21T01:07:37.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:37 smithi160 ceph-mon[34696]: pgmap v512: 129 pgs: 129 active+clean; 88 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 199 op/s 2024-09-21T01:07:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:37 smithi039 ceph-mon[30415]: pgmap v512: 129 pgs: 129 active+clean; 88 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 199 op/s 2024-09-21T01:07:38.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 150435 11.38 MB/sec execute 5 sec latency 14.819 ms 2024-09-21T01:07:39.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 152843 11.78 MB/sec execute 6 sec latency 9.839 ms 2024-09-21T01:07:39.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:39 smithi150 ceph-mon[34945]: pgmap v513: 129 pgs: 129 active+clean; 90 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 237 op/s 2024-09-21T01:07:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:39 smithi160 ceph-mon[34696]: pgmap v513: 129 pgs: 129 active+clean; 90 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 237 op/s 2024-09-21T01:07:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:39 smithi039 ceph-mon[30415]: pgmap v513: 129 pgs: 129 active+clean; 90 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 237 op/s 2024-09-21T01:07:40.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 154799 10.97 MB/sec execute 7 sec latency 12.364 ms 2024-09-21T01:07:41.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 156510 11.26 MB/sec execute 8 sec latency 104.977 ms 2024-09-21T01:07:41.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:41 smithi160 ceph-mon[34696]: pgmap v514: 129 pgs: 129 active+clean; 93 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-09-21T01:07:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:41 smithi039 ceph-mon[30415]: pgmap v514: 129 pgs: 129 active+clean; 93 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-09-21T01:07:42.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:41 smithi150 ceph-mon[34945]: pgmap v514: 129 pgs: 129 active+clean; 93 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-09-21T01:07:42.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 157036 10.10 MB/sec execute 9 sec latency 330.588 ms 2024-09-21T01:07:43.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 159467 10.46 MB/sec execute 10 sec latency 33.162 ms 2024-09-21T01:07:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:43 smithi160 ceph-mon[34696]: pgmap v515: 129 pgs: 129 active+clean; 95 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-21T01:07:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:43 smithi039 ceph-mon[30415]: pgmap v515: 129 pgs: 129 active+clean; 95 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-21T01:07:44.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:43 smithi150 ceph-mon[34945]: pgmap v515: 129 pgs: 129 active+clean; 95 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-09-21T01:07:44.110 INFO:tasks.workunit.client.0.smithi039.stdout: 1 161049 10.39 MB/sec execute 11 sec latency 16.486 ms 2024-09-21T01:07:45.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 163433 10.67 MB/sec execute 12 sec latency 12.699 ms 2024-09-21T01:07:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:45 smithi160 ceph-mon[34696]: pgmap v516: 129 pgs: 129 active+clean; 99 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-09-21T01:07:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:45 smithi039 ceph-mon[30415]: pgmap v516: 129 pgs: 129 active+clean; 99 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-09-21T01:07:46.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:45 smithi150 ceph-mon[34945]: pgmap v516: 129 pgs: 129 active+clean; 99 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-09-21T01:07:46.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 165075 10.29 MB/sec execute 13 sec latency 19.816 ms 2024-09-21T01:07:47.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 165100 9.56 MB/sec execute 14 sec latency 863.817 ms 2024-09-21T01:07:47.828 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:47 smithi039 ceph-mon[30415]: pgmap v517: 129 pgs: 129 active+clean; 100 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-09-21T01:07:47.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:47 smithi160 ceph-mon[34696]: pgmap v517: 129 pgs: 129 active+clean; 100 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-09-21T01:07:47.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:47 smithi150 ceph-mon[34945]: pgmap v517: 129 pgs: 129 active+clean; 100 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-09-21T01:07:48.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 165791 9.09 MB/sec execute 15 sec latency 1037.859 ms 2024-09-21T01:07:49.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 167533 9.25 MB/sec execute 16 sec latency 20.697 ms 2024-09-21T01:07:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:49 smithi160 ceph-mon[34696]: pgmap v518: 129 pgs: 129 active+clean; 107 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-09-21T01:07:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:49 smithi039 ceph-mon[30415]: pgmap v518: 129 pgs: 129 active+clean; 107 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-09-21T01:07:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:49 smithi150 ceph-mon[34945]: pgmap v518: 129 pgs: 129 active+clean; 107 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-09-21T01:07:50.113 INFO:tasks.workunit.client.0.smithi039.stdout: 1 170106 9.53 MB/sec execute 17 sec latency 12.657 ms 2024-09-21T01:07:51.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 171779 9.54 MB/sec execute 18 sec latency 14.393 ms 2024-09-21T01:07:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:51 smithi160 ceph-mon[34696]: pgmap v519: 129 pgs: 129 active+clean; 110 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 211 op/s 2024-09-21T01:07:51.859 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:51 smithi039 ceph-mon[30415]: pgmap v519: 129 pgs: 129 active+clean; 110 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 211 op/s 2024-09-21T01:07:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:51 smithi150 ceph-mon[34945]: pgmap v519: 129 pgs: 129 active+clean; 110 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 211 op/s 2024-09-21T01:07:52.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 172037 9.05 MB/sec execute 19 sec latency 834.041 ms 2024-09-21T01:07:53.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 172037 8.59 MB/sec execute 20 sec latency 1834.102 ms 2024-09-21T01:07:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:53 smithi160 ceph-mon[34696]: pgmap v520: 129 pgs: 129 active+clean; 112 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 227 op/s 2024-09-21T01:07:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:53 smithi039 ceph-mon[30415]: pgmap v520: 129 pgs: 129 active+clean; 112 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 227 op/s 2024-09-21T01:07:54.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:53 smithi150 ceph-mon[34945]: pgmap v520: 129 pgs: 129 active+clean; 112 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 227 op/s 2024-09-21T01:07:54.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 173247 8.53 MB/sec execute 21 sec latency 2384.282 ms 2024-09-21T01:07:55.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 174891 8.53 MB/sec execute 22 sec latency 20.458 ms 2024-09-21T01:07:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:55 smithi160 ceph-mon[34696]: pgmap v521: 129 pgs: 129 active+clean; 112 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-09-21T01:07:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:55 smithi039 ceph-mon[30415]: pgmap v521: 129 pgs: 129 active+clean; 112 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-09-21T01:07:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:55 smithi150 ceph-mon[34945]: pgmap v521: 129 pgs: 129 active+clean; 112 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-09-21T01:07:56.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 177504 8.85 MB/sec execute 23 sec latency 13.423 ms 2024-09-21T01:07:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:56 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:07:56.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:56 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:07:57.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:56 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:07:57.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 177835 8.53 MB/sec execute 24 sec latency 585.292 ms 2024-09-21T01:07:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:57 smithi039 ceph-mon[30415]: pgmap v522: 129 pgs: 129 active+clean; 112 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2024-09-21T01:07:58.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:57 smithi150 ceph-mon[34945]: pgmap v522: 129 pgs: 129 active+clean; 112 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2024-09-21T01:07:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:57 smithi160 ceph-mon[34696]: pgmap v522: 129 pgs: 129 active+clean; 112 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2024-09-21T01:07:58.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 177908 8.19 MB/sec execute 25 sec latency 1521.815 ms 2024-09-21T01:07:58.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:07:58.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:07:58.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:58 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:07:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:07:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:07:59.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:58 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:07:59.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:07:59.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:07:59.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:58 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:07:59.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 180124 8.30 MB/sec execute 26 sec latency 15.122 ms 2024-09-21T01:07:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:07:59 smithi039 ceph-mon[30415]: pgmap v523: 129 pgs: 129 active+clean; 109 MiB data, 959 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 211 op/s 2024-09-21T01:08:00.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:07:59 smithi150 ceph-mon[34945]: pgmap v523: 129 pgs: 129 active+clean; 109 MiB data, 959 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 211 op/s 2024-09-21T01:08:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:07:59 smithi160 ceph-mon[34696]: pgmap v523: 129 pgs: 129 active+clean; 109 MiB data, 959 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 211 op/s 2024-09-21T01:08:00.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 181784 8.32 MB/sec execute 27 sec latency 19.640 ms 2024-09-21T01:08:01.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 184147 8.48 MB/sec execute 28 sec latency 48.194 ms 2024-09-21T01:08:01.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:01 smithi039 ceph-mon[30415]: pgmap v524: 129 pgs: 129 active+clean; 112 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 219 op/s 2024-09-21T01:08:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:01 smithi150 ceph-mon[34945]: pgmap v524: 129 pgs: 129 active+clean; 112 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 219 op/s 2024-09-21T01:08:02.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:01 smithi160 ceph-mon[34696]: pgmap v524: 129 pgs: 129 active+clean; 112 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 219 op/s 2024-09-21T01:08:02.111 INFO:tasks.workunit.client.0.smithi039.stdout: 1 184171 8.21 MB/sec execute 29 sec latency 307.539 ms 2024-09-21T01:08:03.112 INFO:tasks.workunit.client.0.smithi039.stdout: 1 184181 7.94 MB/sec execute 30 sec latency 740.200 ms 2024-09-21T01:08:03.753 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:03 smithi039 ceph-mon[30415]: pgmap v525: 129 pgs: 129 active+clean; 114 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 213 op/s 2024-09-21T01:08:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:03 smithi150 ceph-mon[34945]: pgmap v525: 129 pgs: 129 active+clean; 114 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 213 op/s 2024-09-21T01:08:04.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:03 smithi160 ceph-mon[34696]: pgmap v525: 129 pgs: 129 active+clean; 114 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 213 op/s 2024-09-21T01:08:04.112 INFO:tasks.workunit.client.0.smithi039.stdout: 1 184623 7.80 MB/sec execute 31 sec latency 1478.150 ms 2024-09-21T01:08:05.112 INFO:tasks.workunit.client.0.smithi039.stdout: 1 186232 7.76 MB/sec execute 32 sec latency 22.205 ms 2024-09-21T01:08:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:05 smithi039 ceph-mon[30415]: pgmap v526: 129 pgs: 129 active+clean; 115 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 190 op/s 2024-09-21T01:08:06.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:05 smithi150 ceph-mon[34945]: pgmap v526: 129 pgs: 129 active+clean; 115 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 190 op/s 2024-09-21T01:08:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:05 smithi160 ceph-mon[34696]: pgmap v526: 129 pgs: 129 active+clean; 115 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 190 op/s 2024-09-21T01:08:06.112 INFO:tasks.workunit.client.0.smithi039.stdout: 1 188450 7.94 MB/sec execute 33 sec latency 24.689 ms 2024-09-21T01:08:07.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 189408 7.86 MB/sec execute 34 sec latency 236.254 ms 2024-09-21T01:08:08.069 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:07 smithi150 ceph-mon[34945]: pgmap v527: 129 pgs: 129 active+clean; 117 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-09-21T01:08:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:07 smithi160 ceph-mon[34696]: pgmap v527: 129 pgs: 129 active+clean; 117 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-09-21T01:08:08.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 190436 7.69 MB/sec execute 35 sec latency 141.170 ms 2024-09-21T01:08:08.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:07 smithi039 ceph-mon[30415]: pgmap v527: 129 pgs: 129 active+clean; 117 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-09-21T01:08:09.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 191859 7.80 MB/sec execute 36 sec latency 282.704 ms 2024-09-21T01:08:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:09 smithi150 ceph-mon[34945]: pgmap v528: 129 pgs: 129 active+clean; 116 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 245 op/s 2024-09-21T01:08:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:09 smithi160 ceph-mon[34696]: pgmap v528: 129 pgs: 129 active+clean; 116 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 245 op/s 2024-09-21T01:08:10.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 193751 7.76 MB/sec execute 37 sec latency 43.788 ms 2024-09-21T01:08:10.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:09 smithi039 ceph-mon[30415]: pgmap v528: 129 pgs: 129 active+clean; 116 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 245 op/s 2024-09-21T01:08:11.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:10 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:08:11.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:10 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:08:11.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 195692 7.90 MB/sec execute 38 sec latency 11.282 ms 2024-09-21T01:08:11.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:10 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:08:11.957 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:11 smithi039 ceph-mon[30415]: pgmap v529: 129 pgs: 129 active+clean; 117 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-09-21T01:08:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:11 smithi150 ceph-mon[34945]: pgmap v529: 129 pgs: 129 active+clean; 117 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-09-21T01:08:12.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:11 smithi160 ceph-mon[34696]: pgmap v529: 129 pgs: 129 active+clean; 117 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-09-21T01:08:12.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 196782 7.84 MB/sec execute 39 sec latency 230.762 ms 2024-09-21T01:08:13.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 196782 7.64 MB/sec execute 40 sec latency 1166.512 ms 2024-09-21T01:08:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:13 smithi150 ceph-mon[34945]: pgmap v530: 129 pgs: 129 active+clean; 119 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 228 op/s 2024-09-21T01:08:14.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:13 smithi160 ceph-mon[34696]: pgmap v530: 129 pgs: 129 active+clean; 119 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 228 op/s 2024-09-21T01:08:14.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 198893 7.78 MB/sec execute 41 sec latency 1291.676 ms 2024-09-21T01:08:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:13 smithi039 ceph-mon[30415]: pgmap v530: 129 pgs: 129 active+clean; 119 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 228 op/s 2024-09-21T01:08:15.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 200498 7.73 MB/sec execute 42 sec latency 19.873 ms 2024-09-21T01:08:16.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:15 smithi150 ceph-mon[34945]: pgmap v531: 129 pgs: 129 active+clean; 119 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-09-21T01:08:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:15 smithi160 ceph-mon[34696]: pgmap v531: 129 pgs: 129 active+clean; 119 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-09-21T01:08:16.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 202655 7.87 MB/sec execute 43 sec latency 10.335 ms 2024-09-21T01:08:16.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:15 smithi039 ceph-mon[30415]: pgmap v531: 129 pgs: 129 active+clean; 119 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-09-21T01:08:17.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 203890 7.81 MB/sec execute 44 sec latency 281.931 ms 2024-09-21T01:08:17.957 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:17 smithi039 ceph-mon[30415]: pgmap v532: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 230 op/s 2024-09-21T01:08:18.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:17 smithi150 ceph-mon[34945]: pgmap v532: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 230 op/s 2024-09-21T01:08:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:17 smithi160 ceph-mon[34696]: pgmap v532: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 230 op/s 2024-09-21T01:08:18.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 203890 7.64 MB/sec execute 45 sec latency 1281.994 ms 2024-09-21T01:08:19.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 205395 7.65 MB/sec execute 46 sec latency 1399.747 ms 2024-09-21T01:08:20.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:19 smithi150 ceph-mon[34945]: pgmap v533: 129 pgs: 129 active+clean; 123 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 217 op/s 2024-09-21T01:08:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:19 smithi160 ceph-mon[34696]: pgmap v533: 129 pgs: 129 active+clean; 123 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 217 op/s 2024-09-21T01:08:20.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 207177 7.72 MB/sec execute 47 sec latency 13.053 ms 2024-09-21T01:08:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:19 smithi039 ceph-mon[30415]: pgmap v533: 129 pgs: 129 active+clean; 123 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 217 op/s 2024-09-21T01:08:21.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:20 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:08:21.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:20 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:08:21.124 INFO:tasks.workunit.client.0.smithi039.stdout: 1 209515 7.84 MB/sec execute 48 sec latency 18.143 ms 2024-09-21T01:08:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:20 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:08:21.975 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:21 smithi039 ceph-mon[30415]: pgmap v534: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 198 op/s 2024-09-21T01:08:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:21 smithi150 ceph-mon[34945]: pgmap v534: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 198 op/s 2024-09-21T01:08:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:21 smithi160 ceph-mon[34696]: pgmap v534: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 198 op/s 2024-09-21T01:08:22.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 210837 7.79 MB/sec execute 49 sec latency 137.027 ms 2024-09-21T01:08:23.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 210872 7.64 MB/sec execute 50 sec latency 499.248 ms 2024-09-21T01:08:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:23 smithi150 ceph-mon[34945]: pgmap v535: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 235 op/s 2024-09-21T01:08:24.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:23 smithi160 ceph-mon[34696]: pgmap v535: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 235 op/s 2024-09-21T01:08:24.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 213055 7.76 MB/sec execute 51 sec latency 188.930 ms 2024-09-21T01:08:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:23 smithi039 ceph-mon[30415]: pgmap v535: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 235 op/s 2024-09-21T01:08:25.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 214596 7.72 MB/sec execute 52 sec latency 14.953 ms 2024-09-21T01:08:26.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:25 smithi150 ceph-mon[34945]: pgmap v536: 129 pgs: 129 active+clean; 128 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 208 op/s 2024-09-21T01:08:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:25 smithi160 ceph-mon[34696]: pgmap v536: 129 pgs: 129 active+clean; 128 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 208 op/s 2024-09-21T01:08:26.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 216853 7.83 MB/sec execute 53 sec latency 20.472 ms 2024-09-21T01:08:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:25 smithi039 ceph-mon[30415]: pgmap v536: 129 pgs: 129 active+clean; 128 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 208 op/s 2024-09-21T01:08:27.129 INFO:tasks.workunit.client.0.smithi039.stdout: 1 218840 7.82 MB/sec execute 54 sec latency 51.483 ms 2024-09-21T01:08:27.967 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:27 smithi039 ceph-mon[30415]: pgmap v537: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-09-21T01:08:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:27 smithi150 ceph-mon[34945]: pgmap v537: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-09-21T01:08:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:27 smithi160 ceph-mon[34696]: pgmap v537: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-09-21T01:08:28.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 219351 7.77 MB/sec execute 55 sec latency 700.976 ms 2024-09-21T01:08:29.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 220934 7.77 MB/sec execute 56 sec latency 13.633 ms 2024-09-21T01:08:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:29 smithi150 ceph-mon[34945]: pgmap v538: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 231 op/s 2024-09-21T01:08:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:29 smithi160 ceph-mon[34696]: pgmap v538: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 231 op/s 2024-09-21T01:08:30.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 223618 7.93 MB/sec execute 57 sec latency 12.494 ms 2024-09-21T01:08:30.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:29 smithi039 ceph-mon[30415]: pgmap v538: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 231 op/s 2024-09-21T01:08:31.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 225208 7.91 MB/sec execute 58 sec latency 13.086 ms 2024-09-21T01:08:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:31 smithi150 ceph-mon[34945]: pgmap v539: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 208 op/s 2024-09-21T01:08:32.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:31 smithi160 ceph-mon[34696]: pgmap v539: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 208 op/s 2024-09-21T01:08:32.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 227610 8.01 MB/sec execute 59 sec latency 12.411 ms 2024-09-21T01:08:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:31 smithi039 ceph-mon[30415]: pgmap v539: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 208 op/s 2024-09-21T01:08:33.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 227766 7.88 MB/sec execute 60 sec latency 595.876 ms 2024-09-21T01:08:34.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:33 smithi150 ceph-mon[34945]: pgmap v540: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 213 op/s 2024-09-21T01:08:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:33 smithi160 ceph-mon[34696]: pgmap v540: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 213 op/s 2024-09-21T01:08:34.127 INFO:tasks.workunit.client.0.smithi039.stdout: 1 229984 7.95 MB/sec execute 61 sec latency 687.032 ms 2024-09-21T01:08:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:33 smithi039 ceph-mon[30415]: pgmap v540: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 213 op/s 2024-09-21T01:08:35.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 231677 7.96 MB/sec execute 62 sec latency 14.642 ms 2024-09-21T01:08:36.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:35 smithi150 ceph-mon[34945]: pgmap v541: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 186 op/s 2024-09-21T01:08:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:35 smithi160 ceph-mon[34696]: pgmap v541: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 186 op/s 2024-09-21T01:08:36.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 234287 8.08 MB/sec execute 63 sec latency 12.993 ms 2024-09-21T01:08:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:35 smithi039 ceph-mon[30415]: pgmap v541: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 186 op/s 2024-09-21T01:08:37.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 235851 8.06 MB/sec execute 64 sec latency 16.090 ms 2024-09-21T01:08:38.022 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:37 smithi039 ceph-mon[30415]: pgmap v542: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 225 op/s 2024-09-21T01:08:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:37 smithi150 ceph-mon[34945]: pgmap v542: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 225 op/s 2024-09-21T01:08:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:37 smithi160 ceph-mon[34696]: pgmap v542: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 225 op/s 2024-09-21T01:08:38.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 236598 7.97 MB/sec execute 65 sec latency 684.101 ms 2024-09-21T01:08:39.129 INFO:tasks.workunit.client.0.smithi039.stdout: 1 238363 8.03 MB/sec execute 66 sec latency 719.949 ms 2024-09-21T01:08:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:39 smithi150 ceph-mon[34945]: pgmap v543: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 225 op/s 2024-09-21T01:08:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:39 smithi160 ceph-mon[34696]: pgmap v543: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 225 op/s 2024-09-21T01:08:40.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 240693 8.10 MB/sec execute 67 sec latency 19.315 ms 2024-09-21T01:08:40.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:39 smithi039 ceph-mon[30415]: pgmap v543: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 225 op/s 2024-09-21T01:08:41.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 242241 8.09 MB/sec execute 68 sec latency 16.508 ms 2024-09-21T01:08:42.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:41 smithi150 ceph-mon[34945]: pgmap v544: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-09-21T01:08:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:41 smithi160 ceph-mon[34696]: pgmap v544: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-09-21T01:08:42.128 INFO:tasks.workunit.client.0.smithi039.stdout: 1 244856 8.21 MB/sec execute 69 sec latency 13.021 ms 2024-09-21T01:08:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:41 smithi039 ceph-mon[30415]: pgmap v544: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-09-21T01:08:43.138 INFO:tasks.workunit.client.0.smithi039.stdout: 1 245210 8.11 MB/sec execute 70 sec latency 604.604 ms 2024-09-21T01:08:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:43 smithi039 ceph-mon[30415]: pgmap v545: 129 pgs: 129 active+clean; 144 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 225 op/s 2024-09-21T01:08:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:43 smithi150 ceph-mon[34945]: pgmap v545: 129 pgs: 129 active+clean; 144 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 225 op/s 2024-09-21T01:08:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:43 smithi160 ceph-mon[34696]: pgmap v545: 129 pgs: 129 active+clean; 144 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 225 op/s 2024-09-21T01:08:44.138 INFO:tasks.workunit.client.0.smithi039.stdout: 1 245670 8.01 MB/sec execute 71 sec latency 1212.741 ms 2024-09-21T01:08:45.138 INFO:tasks.workunit.client.0.smithi039.stdout: 1 247994 8.07 MB/sec execute 72 sec latency 74.497 ms 2024-09-21T01:08:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:45 smithi150 ceph-mon[34945]: pgmap v546: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 216 op/s 2024-09-21T01:08:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:45 smithi160 ceph-mon[34696]: pgmap v546: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 216 op/s 2024-09-21T01:08:46.138 INFO:tasks.workunit.client.0.smithi039.stdout: 1 249591 8.07 MB/sec execute 73 sec latency 14.488 ms 2024-09-21T01:08:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:45 smithi039 ceph-mon[30415]: pgmap v546: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 216 op/s 2024-09-21T01:08:47.139 INFO:tasks.workunit.client.0.smithi039.stdout: 1 252097 8.19 MB/sec execute 74 sec latency 11.484 ms 2024-09-21T01:08:47.841 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:47 smithi039 ceph-mon[30415]: pgmap v547: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-09-21T01:08:48.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:47 smithi150 ceph-mon[34945]: pgmap v547: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-09-21T01:08:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:47 smithi160 ceph-mon[34696]: pgmap v547: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-09-21T01:08:48.138 INFO:tasks.workunit.client.0.smithi039.stdout: 1 252660 8.09 MB/sec execute 75 sec latency 477.721 ms 2024-09-21T01:08:49.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:48 smithi150 ceph-mon[34945]: pgmap v548: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 204 op/s 2024-09-21T01:08:49.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:48 smithi160 ceph-mon[34696]: pgmap v548: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 204 op/s 2024-09-21T01:08:49.138 INFO:tasks.workunit.client.0.smithi039.stdout: 1 252858 7.99 MB/sec execute 76 sec latency 1317.316 ms 2024-09-21T01:08:49.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:48 smithi039 ceph-mon[30415]: pgmap v548: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 204 op/s 2024-09-21T01:08:50.139 INFO:tasks.workunit.client.0.smithi039.stdout: 1 255539 8.10 MB/sec execute 77 sec latency 13.475 ms 2024-09-21T01:08:51.139 INFO:tasks.workunit.client.0.smithi039.stdout: 1 257107 8.08 MB/sec execute 78 sec latency 15.141 ms 2024-09-21T01:08:51.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:51 smithi039 ceph-mon[30415]: pgmap v549: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-09-21T01:08:51.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:51 smithi150 ceph-mon[34945]: pgmap v549: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-09-21T01:08:51.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:51 smithi160 ceph-mon[34696]: pgmap v549: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-09-21T01:08:52.139 INFO:tasks.workunit.client.0.smithi039.stdout: 1 259369 8.16 MB/sec execute 79 sec latency 10.053 ms 2024-09-21T01:08:53.142 INFO:tasks.workunit.client.0.smithi039.stdout: 1 260090 8.08 MB/sec execute 80 sec latency 389.052 ms 2024-09-21T01:08:53.189 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:53 smithi039 ceph-mon[30415]: pgmap v550: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 216 op/s 2024-09-21T01:08:53.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:53 smithi150 ceph-mon[34945]: pgmap v550: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 216 op/s 2024-09-21T01:08:53.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:53 smithi160 ceph-mon[34696]: pgmap v550: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 216 op/s 2024-09-21T01:08:54.142 INFO:tasks.workunit.client.0.smithi039.stdout: 1 260673 8.02 MB/sec execute 81 sec latency 671.821 ms 2024-09-21T01:08:54.247 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-09-21T01:08:54.247 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:08:54.678 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:08:55.142 INFO:tasks.workunit.client.0.smithi039.stdout: 1 262950 8.09 MB/sec execute 82 sec latency 15.025 ms 2024-09-21T01:08:55.364 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:55 smithi039 ceph-mon[30415]: pgmap v551: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 213 op/s 2024-09-21T01:08:55.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 -- 172.21.15.39:0/3985315821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109c60 msgr2=0x7f08a4110490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:55.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 --2- 172.21.15.39:0/3985315821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109c60 0x7f08a4110490 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f089c009ca0 tx=0x7f089c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 -- 172.21.15.39:0/3985315821 shutdown_connections 2024-09-21T01:08:55.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 --2- 172.21.15.39:0/3985315821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109c60 0x7f08a4110490 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 --2- 172.21.15.39:0/3985315821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a4109310 0x7f08a4109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 --2- 172.21.15.39:0/3985315821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a41039d0 0x7f08a4103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 -- 172.21.15.39:0/3985315821 >> 172.21.15.39:0/3985315821 conn(0x7f08a40ff440 msgr2=0x7f08a4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:55.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 -- 172.21.15.39:0/3985315821 shutdown_connections 2024-09-21T01:08:55.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.432+0000 7f08a3200640 1 -- 172.21.15.39:0/3985315821 wait complete. 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.433+0000 7f08a3200640 1 Processor -- start 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.433+0000 7f08a3200640 1 -- start start 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a3200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 0x7f08a4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109310 0x7f08a4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a3200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a4109c60 0x7f08a4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4113970 con 0x7f08a4109310 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a3200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41137f0 con 0x7f08a4109c60 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a3200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4113af0 con 0x7f08a41039d0 2024-09-21T01:08:55.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 0x7f08a4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a2800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a4109c60 0x7f08a4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a1e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 0x7f08a4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42180/0 (socket says 172.21.15.39:42180) 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a1e00640 1 -- 172.21.15.39:0/1998933165 learned_addr learned my addr 172.21.15.39:0/1998933165 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.434+0000 7f08a1400640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109310 0x7f08a4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1e00640 1 -- 172.21.15.39:0/1998933165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a4109c60 msgr2=0x7f08a4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1e00640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a4109c60 0x7f08a4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1e00640 1 -- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109310 msgr2=0x7f08a4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1e00640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109310 0x7f08a4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1e00640 1 -- 172.21.15.39:0/1998933165 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a41a8580 con 0x7f08a41039d0 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a2800640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a4109c60 0x7f08a4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1400640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109310 0x7f08a4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:08:55.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a1e00640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 0x7f08a4075710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f089400ea60 tx=0x7f089400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:55.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089400cde0 con 0x7f08a41039d0 2024-09-21T01:08:55.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08940035f0 con 0x7f08a41039d0 2024-09-21T01:08:55.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a41a8870 con 0x7f08a41039d0 2024-09-21T01:08:55.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894010690 con 0x7f08a41039d0 2024-09-21T01:08:55.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.435+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a41a8e00 con 0x7f08a41039d0 2024-09-21T01:08:55.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.437+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a4103de0 con 0x7f08a41039d0 2024-09-21T01:08:55.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.439+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f08940037a0 con 0x7f08a41039d0 2024-09-21T01:08:55.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.440+0000 7f089aa00640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f08880b6700 0x7f08880b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:55.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.441+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f08940e7dd0 con 0x7f08a41039d0 2024-09-21T01:08:55.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.441+0000 7f08a1400640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f08880b6700 0x7f08880b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:55.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.441+0000 7f08a1400640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f08880b6700 0x7f08880b8c10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f08a4102b40 tx=0x7f0890005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:55.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.446+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f08940adf40 con 0x7f08a41039d0 2024-09-21T01:08:55.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:55 smithi150 ceph-mon[34945]: pgmap v551: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 213 op/s 2024-09-21T01:08:55.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:55 smithi160 ceph-mon[34696]: pgmap v551: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 213 op/s 2024-09-21T01:08:55.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.722+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08a402e640 con 0x7f08a41039d0 2024-09-21T01:08:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.723+0000 7f089aa00640 1 -- 172.21.15.39:0/1998933165 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11911 (secure 0 0 0) 0x7f08940b2320 con 0x7f08a41039d0 2024-09-21T01:08:55.724 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:08:55.725 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"btime":"2024-09-21T01:08:19:688675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:08:19.501727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:08:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f08880b6700 msgr2=0x7f08880b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f08880b6700 0x7f08880b8c10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f08a4102b40 tx=0x7f0890005f90 comp rx=0 tx=0).stop 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 msgr2=0x7f08a4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 0x7f08a4075710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f089400ea60 tx=0x7f089400ef20 comp rx=0 tx=0).stop 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 shutdown_connections 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f08880b6700 0x7f08880b8c10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08a4109c60 0x7f08a4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08a4109310 0x7f08a4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 --2- 172.21.15.39:0/1998933165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f08a41039d0 0x7f08a4075710 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 >> 172.21.15.39:0/1998933165 conn(0x7f08a40ff440 msgr2=0x7f08a4108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.726+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 shutdown_connections 2024-09-21T01:08:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:55.727+0000 7f08a3200640 1 -- 172.21.15.39:0/1998933165 wait complete. 2024-09-21T01:08:56.142 INFO:tasks.workunit.client.0.smithi039.stdout: 1 264948 8.08 MB/sec execute 83 sec latency 14.324 ms 2024-09-21T01:08:56.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1998933165' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:08:56.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1998933165' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:08:56.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1998933165' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:08:56.767 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-09-21T01:08:56.767 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-09-21T01:08:57.142 INFO:tasks.workunit.client.0.smithi039.stdout: 1 266771 8.13 MB/sec execute 84 sec latency 20.380 ms 2024-09-21T01:08:57.172 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:08:57.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:57 smithi039 ceph-mon[30415]: pgmap v552: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-09-21T01:08:57.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:57 smithi150 ceph-mon[34945]: pgmap v552: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-09-21T01:08:57.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:57 smithi160 ceph-mon[34696]: pgmap v552: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-09-21T01:08:58.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268016 8.10 MB/sec execute 85 sec latency 307.400 ms 2024-09-21T01:08:58.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 -- 172.21.15.39:0/3755080440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb54076080 msgr2=0x7fcb5410fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 --2- 172.21.15.39:0/3755080440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb54076080 0x7fcb5410fa90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009900 tx=0x7fcb4002a410 comp rx=0 tx=0).stop 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 -- 172.21.15.39:0/3755080440 shutdown_connections 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 --2- 172.21.15.39:0/3755080440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb5410ffd0 0x7fcb541123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 --2- 172.21.15.39:0/3755080440 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb54076080 0x7fcb5410fa90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 --2- 172.21.15.39:0/3755080440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb54075730 0x7fcb54075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 -- 172.21.15.39:0/3755080440 >> 172.21.15.39:0/3755080440 conn(0x7fcb540ff480 msgr2=0x7fcb541018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.159+0000 7fcb53e00640 1 -- 172.21.15.39:0/3755080440 shutdown_connections 2024-09-21T01:08:58.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.160+0000 7fcb53e00640 1 -- 172.21.15.39:0/3755080440 wait complete. 2024-09-21T01:08:58.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 Processor -- start 2024-09-21T01:08:58.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 -- start start 2024-09-21T01:08:58.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 0x7fcb5419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb54076080 0x7fcb5419cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb5410ffd0 0x7fcb541a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54115b60 con 0x7fcb54075730 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541159e0 con 0x7fcb54076080 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.161+0000 7fcb53e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54115ce0 con 0x7fcb5410ffd0 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 0x7fcb5419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 0x7fcb5419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39878/0 (socket says 172.21.15.39:39878) 2024-09-21T01:08:58.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 -- 172.21.15.39:0/1480957465 learned_addr learned my addr 172.21.15.39:0/1480957465 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 -- 172.21.15.39:0/1480957465 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb5410ffd0 msgr2=0x7fcb541a1e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb5410ffd0 0x7fcb541a1e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 -- 172.21.15.39:0/1480957465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb54076080 msgr2=0x7fcb5419cbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb54076080 0x7fcb5419cbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 -- 172.21.15.39:0/1480957465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb541a25c0 con 0x7fcb54075730 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.162+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 0x7fcb5419c650 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400d770 tx=0x7fcb4400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:58.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.163+0000 7fcb4b400640 1 -- 172.21.15.39:0/1480957465 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44017070 con 0x7fcb54075730 2024-09-21T01:08:58.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.163+0000 7fcb4b400640 1 -- 172.21.15.39:0/1480957465 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb440054d0 con 0x7fcb54075730 2024-09-21T01:08:58.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.163+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb541a28b0 con 0x7fcb54075730 2024-09-21T01:08:58.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.163+0000 7fcb4b400640 1 -- 172.21.15.39:0/1480957465 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44004c30 con 0x7fcb54075730 2024-09-21T01:08:58.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.163+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb541a9010 con 0x7fcb54075730 2024-09-21T01:08:58.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.166+0000 7fcb606c4740 1 Processor -- start 2024-09-21T01:08:58.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.166+0000 7fcb606c4740 1 -- start start 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211ee3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211ee3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211ee3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb606c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557211e58ca0 con 0x557211b12980 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb606c4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557211e480d0 con 0x557211b3b840 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb606c4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557211e4bee0 con 0x557211b1ab60 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb52000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211ee3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb52000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211ee3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42210/0 (socket says 172.21.15.39:42210) 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.167+0000 7fcb52000640 1 -- 172.21.15.39:0/3273395197 learned_addr learned my addr 172.21.15.39:0/3273395197 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb52000640 1 -- 172.21.15.39:0/3273395197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 msgr2=0x557211ee3510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb52000640 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211ee3510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb52000640 1 -- 172.21.15.39:0/3273395197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 msgr2=0x557211ee3a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:08:58.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb52000640 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211ee3a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb52000640 1 -- 172.21.15.39:0/3273395197 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557211dfa400 con 0x557211b1ab60 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb52000640 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211ee3f90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb40031440 tx=0x7fcb40009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ea639bfeb9e2b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb4b400640 1 -- 172.21.15.39:0/1480957465 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fcb44004e10 con 0x7fcb54075730 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.168+0000 7fcb3fe00640 1 -- 172.21.15.39:0/3273395197 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4003e030 con 0x557211b1ab60 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.169+0000 7fcb3fe00640 1 -- 172.21.15.39:0/3273395197 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb400035f0 con 0x557211b1ab60 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.169+0000 7fcb3fe00640 1 -- 172.21.15.39:0/3273395197 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40035040 con 0x557211b1ab60 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 -- 172.21.15.39:0/3273395197 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 msgr2=0x557211ee3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:58.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211ee3f90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb40031440 tx=0x7fcb40009600 comp rx=0 tx=0).stop 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 -- 172.21.15.39:0/3273395197 shutdown_connections 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211ee3f90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211ee3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 --2- 172.21.15.39:0/3273395197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211ee3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb606c4740 1 -- 172.21.15.39:0/3273395197 >> 172.21.15.39:0/3273395197 conn(0x557211b2f990 msgr2=0x557211ee1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb4b400640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcb380b6670 0x7fcb380b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb4b400640 1 -- 172.21.15.39:0/1480957465 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcb440e7620 con 0x7fcb54075730 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.170+0000 7fcb52000640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcb380b6670 0x7fcb380b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.171+0000 7fcb606c4740 1 -- 172.21.15.39:0/3273395197 shutdown_connections 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.171+0000 7fcb606c4740 1 -- 172.21.15.39:0/3273395197 wait complete. 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.171+0000 7fcb52000640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcb380b6670 0x7fcb380b8b80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb40005eb0 tx=0x7fcb40000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:58.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.171+0000 7fcb606c4740 1 -- start start 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 Processor -- start 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211eeec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211efaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557211ca97a0 con 0x557211b3b840 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557211deda10 con 0x557211b12980 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557211ef1c90 con 0x557211b1ab60 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb606c4740 10 client.0 _ll_register_callbacks cb 0x557211dfdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.172+0000 7fcb52000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eef160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42212/0 (socket says 172.21.15.39:42212) 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 -- 172.21.15.39:0/454454815 learned_addr learned my addr 172.21.15.39:0/454454815 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:08:58.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb53400640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211efaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52a00640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211eeec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 -- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 msgr2=0x557211eeec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211eeec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 -- 172.21.15.39:0/454454815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 msgr2=0x557211efaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211efaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.173+0000 7fcb52000640 1 -- 172.21.15.39:0/454454815 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557211bb5b90 con 0x557211b1ab60 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb52000640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eef160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb400037d0 tx=0x7fcb40009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce9ac5076b932f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb3cc00640 1 -- 172.21.15.39:0/454454815 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40055020 con 0x557211b1ab60 2024-09-21T01:08:58.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb400035f0 con 0x557211b1ab60 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb3cc00640 1 -- 172.21.15.39:0/454454815 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcb400314a0 con 0x557211b1ab60 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb3cc00640 1 -- 172.21.15.39:0/454454815 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcb4003e040 con 0x557211b1ab60 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb606c4740 10 client.25702 fetch_fsmap learned FSMap version 30 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.174+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557211efd9a0 con 0x557211b1ab60 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb3cc00640 1 -- 172.21.15.39:0/454454815 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb400035f0 con 0x557211b1ab60 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb3cc00640 1 -- 172.21.15.39:0/454454815 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fcb400516f0 con 0x557211b1ab60 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb606c4740 10 client.25702 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb606c4740 10 client.25702 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb606c4740 20 client.25702 populate_metadata read hostname 'smithi039' 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb606c4740 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211afa1c0 0x557211f00d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb606c4740 4 client.25702 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557211e5abf0 con 0x557211afa1c0 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.175+0000 7fcb52a00640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211afa1c0 0x557211f00d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.176+0000 7fcb52a00640 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211afa1c0 0x557211f00d00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb05cc486739e9ab server_cookie=99dac3946b53efc9 in_seq=0 out_seq=0 2024-09-21T01:08:58.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.176+0000 7fcb3cc00640 10 client.25702 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb3cc00640 1 -- 172.21.15.39:0/454454815 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557211e5abf0 con 0x557211afa1c0 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb3cc00640 10 client.25702 handle_command_reply: tid=0 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 client.25702 shutdown 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 msgr2=0x557211eef160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eef160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb400037d0 tx=0x7fcb40009260 comp rx=0 tx=0).stop 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 shutdown_connections 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211afa1c0 0x557211f00d00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211efaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eef160 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 --2- 172.21.15.39:0/454454815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211eeec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb3cc00640 0 client.25702 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:08:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 >> 172.21.15.39:0/454454815 conn(0x557211b2f990 msgr2=0x557211ee66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:58.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 shutdown_connections 2024-09-21T01:08:58.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.177+0000 7fcb606c4740 1 -- 172.21.15.39:0/454454815 wait complete. 2024-09-21T01:08:58.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.178+0000 7fcb606c4740 20 client.25702 trim_cache size 0 max 16384 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.209+0000 7fcb606c4740 1 Processor -- start 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.209+0000 7fcb606c4740 1 -- start start 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211ef09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eebe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211eec350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb606c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541b5870 con 0x557211b3b840 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb606c4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541b3140 con 0x557211b12980 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb606c4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541b0f90 con 0x557211b1ab60 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb52a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211ef09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb52a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211ef09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:51512/0 (socket says 172.21.15.39:51512) 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb52a00640 1 -- 172.21.15.39:0/1841396629 learned_addr learned my addr 172.21.15.39:0/1841396629 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb53400640 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211eec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.210+0000 7fcb52000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb53400640 1 -- 172.21.15.39:0/1841396629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 msgr2=0x557211eebe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:08:58.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb53400640 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eebe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb53400640 1 -- 172.21.15.39:0/1841396629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 msgr2=0x557211ef09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb53400640 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211ef09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb53400640 1 -- 172.21.15.39:0/1841396629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557211efb660 con 0x557211b3b840 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211ef09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.211+0000 7fcb53400640 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211eec350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009d60 tx=0x7fcb4c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1388e190b4598e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.212+0000 7fcb3f400640 1 -- 172.21.15.39:0/1841396629 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c004d20 con 0x557211b3b840 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.212+0000 7fcb3f400640 1 -- 172.21.15.39:0/1841396629 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb4c009e50 con 0x557211b3b840 2024-09-21T01:08:58.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.212+0000 7fcb3f400640 1 -- 172.21.15.39:0/1841396629 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c005020 con 0x557211b3b840 2024-09-21T01:08:58.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 -- 172.21.15.39:0/1841396629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 msgr2=0x557211eec350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:58.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211eec350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009d60 tx=0x7fcb4c011810 comp rx=0 tx=0).stop 2024-09-21T01:08:58.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 -- 172.21.15.39:0/1841396629 shutdown_connections 2024-09-21T01:08:58.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b3b840 0x557211eec350 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211eebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1841396629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b12980 0x557211ef09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 -- 172.21.15.39:0/1841396629 >> 172.21.15.39:0/1841396629 conn(0x557211b2f990 msgr2=0x557211ee4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 -- 172.21.15.39:0/1841396629 shutdown_connections 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.213+0000 7fcb606c4740 1 -- 172.21.15.39:0/1841396629 wait complete. 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.214+0000 7fcb606c4740 1 -- start start 2024-09-21T01:08:58.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.214+0000 7fcb606c4740 1 Processor -- start 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211f02760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211f02d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211f644c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541b1600 con 0x557211b12980 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541b2a70 con 0x557211b3b840 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541b1cd0 con 0x557211b1ab60 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.215+0000 7fcb606c4740 10 client.0 _ll_register_callbacks cb 0x7fcb40009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211f644c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211f644c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:51524/0 (socket says 172.21.15.39:51524) 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 -- 172.21.15.39:0/1740134418 learned_addr learned my addr 172.21.15.39:0/1740134418 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:08:58.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211f02760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb52000640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211f02d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 -- 172.21.15.39:0/1740134418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 msgr2=0x557211f02d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211f02d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 -- 172.21.15.39:0/1740134418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 msgr2=0x557211f02760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211f02760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 -- 172.21.15.39:0/1740134418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557211f64b30 con 0x557211b3b840 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb52000640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211f02d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.216+0000 7fcb53400640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211f644c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c01b040 tx=0x7fcb4c004850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58484e3584ed5d1e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:08:58.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:08:58.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb3cc00640 1 -- 172.21.15.39:0/1740134418 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c029070 con 0x557211b3b840 2024-09-21T01:08:58.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb3cc00640 1 -- 172.21.15.39:0/1740134418 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcb4c003970 con 0x557211b3b840 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb4c009e50 con 0x557211b3b840 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb3cc00640 1 -- 172.21.15.39:0/1740134418 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcb4c035050 con 0x557211b3b840 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb606c4740 10 client.25718 fetch_fsmap learned FSMap version 30 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.217+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557211f6bf00 con 0x557211b3b840 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb3cc00640 1 -- 172.21.15.39:0/1740134418 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb4c009e50 con 0x557211b3b840 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb3cc00640 1 -- 172.21.15.39:0/1740134418 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fcb4c0057a0 con 0x557211b3b840 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb606c4740 10 client.25718 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb606c4740 10 client.25718 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb606c4740 20 client.25718 populate_metadata read hostname 'smithi039' 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211acf570 0x557211f6e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:08:58.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb606c4740 4 client.25718 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-09-21T01:08:58.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.218+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x557211e566a0 con 0x557211acf570 2024-09-21T01:08:58.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.219+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211acf570 0x557211f6e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:08:58.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.219+0000 7fcb3cc00640 10 client.25718 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:08:58.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:58.219+0000 7fcb52a00640 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211acf570 0x557211f6e5f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=139553b312ff207b server_cookie=24f8f56a158cd7dd in_seq=0 out_seq=0 2024-09-21T01:08:59.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268016 8.00 MB/sec execute 86 sec latency 1307.837 ms 2024-09-21T01:08:59.179 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:59 smithi039 ceph-mon[30415]: pgmap v553: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-09-21T01:08:59.180 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:08:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.523+0000 7fcb3cc00640 1 -- 172.21.15.39:0/1740134418 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557211e566a0 con 0x557211acf570 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.523+0000 7fcb3cc00640 10 client.25718 handle_command_reply: tid=0 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 client.25718 shutdown 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 msgr2=0x557211f644c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211f644c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c01b040 tx=0x7fcb4c004850 comp rx=0 tx=0).stop 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 shutdown_connections 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557211acf570 0x557211f6e5f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557211b3b840 0x557211f644c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557211b1ab60 0x557211f02d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 --2- 172.21.15.39:0/1740134418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557211b12980 0x557211f02760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 >> 172.21.15.39:0/1740134418 conn(0x557211b2f990 msgr2=0x557211ee4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb3cc00640 0 client.25718 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 shutdown_connections 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 1 -- 172.21.15.39:0/1740134418 wait complete. 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.524+0000 7fcb606c4740 20 client.25718 trim_cache size 0 max 16384 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 30, 2024-09-21T01:08:59.527 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 18, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "72706f34": { 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 18, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 130.0, 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:08:59.528 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:08:59.529 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcb380b6670 msgr2=0x7fcb380b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcb380b6670 0x7fcb380b8b80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb40005eb0 tx=0x7fcb40000bc0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 msgr2=0x7fcb5419c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 0x7fcb5419c650 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400d770 tx=0x7fcb4400dc30 comp rx=0 tx=0).stop 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 shutdown_connections 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb5410ffd0 0x7fcb541a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcb380b6670 0x7fcb380b8b80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb54076080 0x7fcb5419cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 --2- 172.21.15.39:0/1480957465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb54075730 0x7fcb5419c650 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.525+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 >> 172.21.15.39:0/1480957465 conn(0x7fcb540ff480 msgr2=0x7fcb54101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.526+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 shutdown_connections 2024-09-21T01:08:59.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:08:59.526+0000 7fcb53e00640 1 -- 172.21.15.39:0/1480957465 wait complete. 2024-09-21T01:08:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:59 smithi150 ceph-mon[34945]: pgmap v553: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-09-21T01:08:59.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:08:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:08:59.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:59 smithi160 ceph-mon[34696]: pgmap v553: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-09-21T01:08:59.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:08:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:09:00.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.91 MB/sec execute 87 sec latency 1643.558 ms 2024-09-21T01:09:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:09:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:09:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:09:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:09:00.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:09:00.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:09:00.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:09:00.809 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '72706f34', quiesce took 6.6 seconds. Will release after: 3.4000000000000004 2024-09-21T01:09:01.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.82 MB/sec execute 88 sec latency 1619.795 ms 2024-09-21T01:09:01.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:01 smithi150 ceph-mon[34945]: pgmap v554: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 201 op/s 2024-09-21T01:09:01.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:01 smithi160 ceph-mon[34696]: pgmap v554: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 201 op/s 2024-09-21T01:09:01.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:01 smithi039 ceph-mon[30415]: pgmap v554: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 201 op/s 2024-09-21T01:09:02.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.73 MB/sec execute 89 sec latency 2619.878 ms 2024-09-21T01:09:03.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.65 MB/sec execute 90 sec latency 3619.929 ms 2024-09-21T01:09:03.350 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:03 smithi039 ceph-mon[30415]: pgmap v555: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.3 MiB/s wr, 183 op/s 2024-09-21T01:09:03.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:03 smithi150 ceph-mon[34945]: pgmap v555: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.3 MiB/s wr, 183 op/s 2024-09-21T01:09:03.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:03 smithi160 ceph-mon[34696]: pgmap v555: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.3 MiB/s wr, 183 op/s 2024-09-21T01:09:04.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.56 MB/sec execute 91 sec latency 4620.000 ms 2024-09-21T01:09:04.210 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:09:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:05.143 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.48 MB/sec execute 92 sec latency 5620.119 ms 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 -- 172.21.15.39:0/3719989425 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1394103b60 msgr2=0x7f1394103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 --2- 172.21.15.39:0/3719989425 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1394103b60 0x7f1394103f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1384009900 tx=0x7f138402a410 comp rx=0 tx=0).stop 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 -- 172.21.15.39:0/3719989425 shutdown_connections 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 --2- 172.21.15.39:0/3719989425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394109df0 0x7f1394110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 --2- 172.21.15.39:0/3719989425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13941094a0 0x7f13941098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 --2- 172.21.15.39:0/3719989425 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1394103b60 0x7f1394103f70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 -- 172.21.15.39:0/3719989425 >> 172.21.15.39:0/3719989425 conn(0x7f139407b850 msgr2=0x7f139407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:05.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 -- 172.21.15.39:0/3719989425 shutdown_connections 2024-09-21T01:09:05.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.398+0000 7f139a600640 1 -- 172.21.15.39:0/3719989425 wait complete. 2024-09-21T01:09:05.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.399+0000 7f139a600640 1 Processor -- start 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 -- start start 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394103b60 0x7f1394075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 0x7f1394075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1394109df0 0x7f1394076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1394113ab0 con 0x7f1394109df0 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1394113930 con 0x7f1394103b60 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f139a600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1394113c30 con 0x7f13941094a0 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f1393400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394103b60 0x7f1394075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f1393e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1394109df0 0x7f1394076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f138aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 0x7f1394075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f1393400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394103b60 0x7f1394075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:54158/0 (socket says 172.21.15.39:54158) 2024-09-21T01:09:05.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f1393e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1394109df0 0x7f1394076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58016/0 (socket says 172.21.15.39:58016) 2024-09-21T01:09:05.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f1393400640 1 -- 172.21.15.39:0/1610621626 learned_addr learned my addr 172.21.15.39:0/1610621626 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:05.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.400+0000 7f138aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 0x7f1394075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48658/0 (socket says 172.21.15.39:48658) 2024-09-21T01:09:05.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f138aa00640 1 -- 172.21.15.39:0/1610621626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394103b60 msgr2=0x7f1394075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:05.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f138aa00640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394103b60 0x7f1394075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f138aa00640 1 -- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1394109df0 msgr2=0x7f1394076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f138aa00640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1394109df0 0x7f1394076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f138aa00640 1 -- 172.21.15.39:0/1610621626 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13941a8740 con 0x7f13941094a0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f138aa00640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 0x7f1394075c50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f139406c9f0 tx=0x7f137c00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c010430 con 0x7f13941094a0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f137c010a80 con 0x7f13941094a0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.401+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13941a8a30 con 0x7f13941094a0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.402+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c017a20 con 0x7f13941094a0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.402+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13941a8f70 con 0x7f13941094a0 2024-09-21T01:09:05.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.403+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1394103f70 con 0x7f13941094a0 2024-09-21T01:09:05.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.405+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f137c017be0 con 0x7f13941094a0 2024-09-21T01:09:05.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.407+0000 7f1390c00640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13780b6680 0x7f13780b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:05.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.408+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f137c0e8500 con 0x7f13941094a0 2024-09-21T01:09:05.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.408+0000 7f1393400640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13780b6680 0x7f13780b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:05.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.408+0000 7f1393400640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13780b6680 0x7f13780b8b90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1384009230 tx=0x7f13840058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:05.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.409+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f137c0ae700 con 0x7f13941094a0 2024-09-21T01:09:05.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:05 smithi150 ceph-mon[34945]: pgmap v556: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-09-21T01:09:05.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:05 smithi160 ceph-mon[34696]: pgmap v556: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-09-21T01:09:05.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.645+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1394076a10 con 0x7f13941094a0 2024-09-21T01:09:05.647 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:05 smithi039 ceph-mon[30415]: pgmap v556: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-09-21T01:09:05.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.646+0000 7f1390c00640 1 -- 172.21.15.39:0/1610621626 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11911 (secure 0 0 0) 0x7f1394076a10 con 0x7f13941094a0 2024-09-21T01:09:05.648 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:09:05.648 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"btime":"2024-09-21T01:08:19:688675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:08:19.501727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:09:05.648 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-09-21T01:09:05.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13780b6680 msgr2=0x7f13780b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:05.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13780b6680 0x7f13780b8b90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1384009230 tx=0x7f13840058d0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 msgr2=0x7f1394075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:05.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 0x7f1394075c50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f139406c9f0 tx=0x7f137c00ef90 comp rx=0 tx=0).stop 2024-09-21T01:09:05.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 shutdown_connections 2024-09-21T01:09:05.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1394109df0 0x7f1394076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13941094a0 0x7f1394075c50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13780b6680 0x7f13780b8b90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 --2- 172.21.15.39:0/1610621626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1394103b60 0x7f1394075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:05.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.649+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 >> 172.21.15.39:0/1610621626 conn(0x7f139407b850 msgr2=0x7f1394107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:05.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.650+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 shutdown_connections 2024-09-21T01:09:05.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:05.650+0000 7f139a600640 1 -- 172.21.15.39:0/1610621626 wait complete. 2024-09-21T01:09:06.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.40 MB/sec execute 93 sec latency 6620.649 ms 2024-09-21T01:09:06.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:06 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1610621626' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:06.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:06 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1610621626' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:06.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:06 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1610621626' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:06.826 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 72706f34 --release --await' 2024-09-21T01:09:06.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 72706f34 --release --await 2024-09-21T01:09:07.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268027 7.32 MB/sec execute 94 sec latency 7620.688 ms 2024-09-21T01:09:07.213 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:07.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:07 smithi039 ceph-mon[30415]: pgmap v557: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 141 op/s 2024-09-21T01:09:07.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:07 smithi150 ceph-mon[34945]: pgmap v557: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 141 op/s 2024-09-21T01:09:07.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:07 smithi160 ceph-mon[34696]: pgmap v557: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 141 op/s 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 -- 172.21.15.39:0/1872829569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 msgr2=0x7f6758103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 --2- 172.21.15.39:0/1872829569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f6758103f90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6748009900 tx=0x7f674802a410 comp rx=0 tx=0).stop 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 -- 172.21.15.39:0/1872829569 shutdown_connections 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 --2- 172.21.15.39:0/1872829569 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6758109e10 0x7f6758110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 --2- 172.21.15.39:0/1872829569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 0x7f67581098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 --2- 172.21.15.39:0/1872829569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f6758103f90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 -- 172.21.15.39:0/1872829569 >> 172.21.15.39:0/1872829569 conn(0x7f67580ff5d0 msgr2=0x7f6758101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 -- 172.21.15.39:0/1872829569 shutdown_connections 2024-09-21T01:09:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.899+0000 7f675da00640 1 -- 172.21.15.39:0/1872829569 wait complete. 2024-09-21T01:09:07.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.900+0000 7f675da00640 1 Processor -- start 2024-09-21T01:09:07.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.900+0000 7f675da00640 1 -- start start 2024-09-21T01:09:07.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f675da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f675819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f675da00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 0x7f675819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f675da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6758109e10 0x7f675819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f675da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6758113ad0 con 0x7f6758103b80 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f675da00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6758113950 con 0x7f67581094c0 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f675da00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6758113c50 con 0x7f6758109e10 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f6756a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f675819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f6756a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f675819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58034/0 (socket says 172.21.15.39:58034) 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f6756a00640 1 -- 172.21.15.39:0/1216657677 learned_addr learned my addr 172.21.15.39:0/1216657677 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f674e000640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 0x7f675819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.901+0000 7f6757400640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6758109e10 0x7f675819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674e000640 1 -- 172.21.15.39:0/1216657677 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6758109e10 msgr2=0x7f675819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674e000640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6758109e10 0x7f675819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674e000640 1 -- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 msgr2=0x7f675819a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674e000640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f675819a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674e000640 1 -- 172.21.15.39:0/1216657677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67581a03f0 con 0x7f67581094c0 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f6756a00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f675819a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:09:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674e000640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 0x7f675819aa20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f675806ca10 tx=0x7f674000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674fe00640 1 -- 172.21.15.39:0/1216657677 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674000cc40 con 0x7f67581094c0 2024-09-21T01:09:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67581a06e0 con 0x7f67581094c0 2024-09-21T01:09:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.902+0000 7f674fe00640 1 -- 172.21.15.39:0/1216657677 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674000cde0 con 0x7f67581094c0 2024-09-21T01:09:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.903+0000 7f674fe00640 1 -- 172.21.15.39:0/1216657677 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6740018760 con 0x7f67581094c0 2024-09-21T01:09:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.903+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67581a6e40 con 0x7f67581094c0 2024-09-21T01:09:07.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 Processor -- start 2024-09-21T01:09:07.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 -- start start 2024-09-21T01:09:07.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b00351c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0035700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc6d840 0x55d4b0035c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d4affac290 con 0x55d4afc6d840 2024-09-21T01:09:07.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d4aff9b6c0 con 0x55d4afc65660 2024-09-21T01:09:07.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.906+0000 7f6764617740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d4aff9f4d0 con 0x55d4afc8e520 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f674fe00640 1 -- 172.21.15.39:0/1216657677 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6740010a90 con 0x7f67581094c0 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b00351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6756a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0035700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b00351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48686/0 (socket says 172.21.15.39:48686) 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 -- 172.21.15.39:0/2634089683 learned_addr learned my addr 172.21.15.39:0/2634089683 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f674e000640 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc6d840 0x55d4b0035c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 -- 172.21.15.39:0/2634089683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 msgr2=0x55d4b0035700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0035700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 -- 172.21.15.39:0/2634089683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc6d840 msgr2=0x55d4b0035c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc6d840 0x55d4b0035c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6757400640 1 -- 172.21.15.39:0/2634089683 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4aff28460 con 0x55d4afc8e520 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f6756a00640 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0035700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.907+0000 7f674e000640 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc6d840 0x55d4b0035c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.908+0000 7f6757400640 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b00351c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6750011440 tx=0x7f6750011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce91b11143bc464a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.908+0000 7f6747e00640 1 -- 172.21.15.39:0/2634089683 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6750004d20 con 0x55d4afc8e520 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.908+0000 7f6747e00640 1 -- 172.21.15.39:0/2634089683 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6750004050 con 0x55d4afc8e520 2024-09-21T01:09:07.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.908+0000 7f6747e00640 1 -- 172.21.15.39:0/2634089683 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675000f7a0 con 0x55d4afc8e520 2024-09-21T01:09:07.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f674fe00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f673c0b6680 0x7f673c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f674fe00640 1 -- 172.21.15.39:0/1216657677 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f67400e7d20 con 0x7f67581094c0 2024-09-21T01:09:07.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 -- 172.21.15.39:0/2634089683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 msgr2=0x55d4b00351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b00351c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6750011440 tx=0x7f6750011900 comp rx=0 tx=0).stop 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6756a00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f673c0b6680 0x7f673c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 -- 172.21.15.39:0/2634089683 shutdown_connections 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc6d840 0x55d4b0035c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0035700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 --2- 172.21.15.39:0/2634089683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b00351c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 -- 172.21.15.39:0/2634089683 >> 172.21.15.39:0/2634089683 conn(0x55d4afc82910 msgr2=0x55d4b00330b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 -- 172.21.15.39:0/2634089683 shutdown_connections 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.909+0000 7f6764617740 1 -- 172.21.15.39:0/2634089683 wait complete. 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.910+0000 7f6756a00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f673c0b6680 0x7f673c0b8b90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6748031440 tx=0x7f67480058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.910+0000 7f6764617740 1 -- start start 2024-09-21T01:09:07.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.910+0000 7f6764617740 1 Processor -- start 2024-09-21T01:09:07.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b00451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0045700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0051000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d4afdfcbb0 con 0x55d4afc8e520 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d4aff366c0 con 0x55d4afc65660 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d4b0047cd0 con 0x55d4afc6d840 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6764617740 10 client.0 _ll_register_callbacks cb 0x55d4aff51490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6757400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0051000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f674e000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0045700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6756a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b00451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6757400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0051000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58048/0 (socket says 172.21.15.39:58048) 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f6757400640 1 -- 172.21.15.39:0/2271553915 learned_addr learned my addr 172.21.15.39:0/2271553915 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f674e000640 1 -- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 msgr2=0x55d4b00451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f674e000640 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b00451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f674e000640 1 -- 172.21.15.39:0/2271553915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 msgr2=0x55d4b0051000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f674e000640 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0051000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.911+0000 7f674e000640 1 -- 172.21.15.39:0/2271553915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4afcf9b00 con 0x55d4afc6d840 2024-09-21T01:09:07.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f674e000640 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0045700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f67400e50f0 tx=0x7f67400aced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb218212bd421fa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6744c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6744c00640 1 -- 172.21.15.39:0/2271553915 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67400ad010 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6750004050 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6744c00640 1 -- 172.21.15.39:0/2271553915 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f67400ac780 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6744c00640 1 -- 172.21.15.39:0/2271553915 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f67400ad3b0 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6764617740 10 client.25723 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4afd70960 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.912+0000 7f6744c00640 1 -- 172.21.15.39:0/2271553915 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6750004050 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6744c00640 1 -- 172.21.15.39:0/2271553915 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f67400b6950 con 0x55d4afc6d840 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6764617740 10 client.25723 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6764617740 10 client.25723 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6764617740 20 client.25723 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6764617740 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc4cea0 0x55d4b0056f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6764617740 4 client.25723 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:09:07.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4affae1e0 con 0x55d4afc4cea0 2024-09-21T01:09:07.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.913+0000 7f6756a00640 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc4cea0 0x55d4b0056f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.914+0000 7f6756a00640 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc4cea0 0x55d4b0056f30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70d22cbea9a82689 server_cookie=4758b19d0287fe2a in_seq=0 out_seq=0 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.914+0000 7f6744c00640 10 client.25723 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6744c00640 1 -- 172.21.15.39:0/2271553915 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d4affae1e0 con 0x55d4afc4cea0 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6744c00640 10 client.25723 handle_command_reply: tid=0 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 client.25723 shutdown 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 msgr2=0x55d4b0045700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0045700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f67400e50f0 tx=0x7f67400aced0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 shutdown_connections 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc4cea0 0x55d4b0056f30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0051000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0045700 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 --2- 172.21.15.39:0/2271553915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b00451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6744c00640 0 client.25723 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 >> 172.21.15.39:0/2271553915 conn(0x55d4afc82910 msgr2=0x55d4b0038350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.915+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 shutdown_connections 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.916+0000 7f6764617740 1 -- 172.21.15.39:0/2271553915 wait complete. 2024-09-21T01:09:07.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.916+0000 7f6764617740 20 client.25723 trim_cache size 0 max 16384 2024-09-21T01:09:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 Processor -- start 2024-09-21T01:09:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 -- start start 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0050f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0051440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0038350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f67581b14e0 con 0x55d4afc8e520 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67581aedb0 con 0x55d4afc65660 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.934+0000 7f6764617740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f67581acc00 con 0x55d4afc6d840 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0038350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0038350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58054/0 (socket says 172.21.15.39:58054) 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 -- 172.21.15.39:0/1106610301 learned_addr learned my addr 172.21.15.39:0/1106610301 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6756a00640 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0050f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f674e000640 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0051440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 -- 172.21.15.39:0/1106610301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 msgr2=0x55d4b0051440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0051440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 -- 172.21.15.39:0/1106610301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 msgr2=0x55d4b0050f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0050f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 -- 172.21.15.39:0/1106610301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b0051980 con 0x55d4afc8e520 2024-09-21T01:09:07.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.935+0000 7f6757400640 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0038350 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6750009f70 tx=0x7f6750003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c27224a09f5ba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:07.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.936+0000 7f6747400640 1 -- 172.21.15.39:0/1106610301 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675001d040 con 0x55d4afc8e520 2024-09-21T01:09:07.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.936+0000 7f6747400640 1 -- 172.21.15.39:0/1106610301 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6750004e50 con 0x55d4afc8e520 2024-09-21T01:09:07.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.936+0000 7f6747400640 1 -- 172.21.15.39:0/1106610301 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67500047d0 con 0x55d4afc8e520 2024-09-21T01:09:07.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.936+0000 7f6764617740 1 -- 172.21.15.39:0/1106610301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 msgr2=0x55d4b0038350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0038350 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6750009f70 tx=0x7f6750003ae0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 -- 172.21.15.39:0/1106610301 shutdown_connections 2024-09-21T01:09:07.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc8e520 0x55d4b0038350 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc6d840 0x55d4b0051440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 --2- 172.21.15.39:0/1106610301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc65660 0x55d4b0050f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 -- 172.21.15.39:0/1106610301 >> 172.21.15.39:0/1106610301 conn(0x55d4afc82910 msgr2=0x55d4b0043840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 -- 172.21.15.39:0/1106610301 shutdown_connections 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 -- 172.21.15.39:0/1106610301 wait complete. 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 -- start start 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.937+0000 7f6764617740 1 Processor -- start 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 0x55d4b00bacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc6d840 0x55d4b00bb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b0049850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f67581ad270 con 0x55d4afc65660 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67581ae6e0 con 0x55d4afc6d840 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f67581ad940 con 0x55d4afc8e520 2024-09-21T01:09:07.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6764617740 10 client.0 _ll_register_callbacks cb 0x7f67400a9ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:07.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6756a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 0x55d4b00bacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6756a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 0x55d4b00bacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58062/0 (socket says 172.21.15.39:58062) 2024-09-21T01:09:07.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6756a00640 1 -- 172.21.15.39:0/1117200579 learned_addr learned my addr 172.21.15.39:0/1117200579 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:07.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f674e000640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc6d840 0x55d4b00bb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.938+0000 7f6757400640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b0049850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f674e000640 1 -- 172.21.15.39:0/1117200579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 msgr2=0x55d4b0049850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f674e000640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b0049850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f674e000640 1 -- 172.21.15.39:0/1117200579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 msgr2=0x55d4b00bacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:07.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f674e000640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 0x55d4b00bacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f674e000640 1 -- 172.21.15.39:0/1117200579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b0049fc0 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6756a00640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 0x55d4b00bacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f674e000640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc6d840 0x55d4b00bb2b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f67400aab10 tx=0x7f67400109e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d59a4f1bb5199db0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6744c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6744c00640 1 -- 172.21.15.39:0/1117200579 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67400d2030 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6750004e50 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6744c00640 1 -- 172.21.15.39:0/1117200579 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f67400aed30 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6744c00640 1 -- 172.21.15.39:0/1117200579 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f67400ad010 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.939+0000 7f6764617740 10 client.25742 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4b00c24e0 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6744c00640 1 -- 172.21.15.39:0/1117200579 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6750004e50 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6744c00640 1 -- 172.21.15.39:0/1117200579 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f67400b6750 con 0x55d4afc6d840 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 10 client.25742 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 10 client.25742 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:07.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 20 client.25742 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:07.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc22250 0x55d4b00c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:07.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 4 client.25742 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "72706f34", "release": true, "await": true}] 2024-09-21T01:09:07.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "72706f34", "release": true, "await": true}) -- 0x55d4affa9c90 con 0x55d4afc22250 2024-09-21T01:09:07.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.940+0000 7f6756a00640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc22250 0x55d4b00c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:07.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.941+0000 7f6756a00640 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc22250 0x55d4b00c4ae0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c378a57d3d9c681 server_cookie=4374b055b5f80501 in_seq=0 out_seq=0 2024-09-21T01:09:07.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.941+0000 7f6744c00640 10 client.25742 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6744c00640 1 -- 172.21.15.39:0/1117200579 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d4affa9c90 con 0x55d4afc22250 2024-09-21T01:09:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6744c00640 10 client.25742 handle_command_reply: tid=0 2024-09-21T01:09:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 client.25742 shutdown 2024-09-21T01:09:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc6d840 msgr2=0x55d4b00bb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc6d840 0x55d4b00bb2b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f67400aab10 tx=0x7f67400109e0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 shutdown_connections 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d4afc22250 0x55d4b00c4ae0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d4afc8e520 0x55d4b0049850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d4afc6d840 0x55d4b00bb2b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 --2- 172.21.15.39:0/1117200579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d4afc65660 0x55d4b00bacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6744c00640 0 client.25742 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 >> 172.21.15.39:0/1117200579 conn(0x55d4afc82910 msgr2=0x55d4b0038350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 shutdown_connections 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 1 -- 172.21.15.39:0/1117200579 wait complete. 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.949+0000 7f6764617740 20 client.25742 trim_cache size 0 max 16384 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 30, 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 20, 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:09:07.952 INFO:teuthology.orchestra.run.smithi039.stdout: "72706f34": { 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 20, 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 130.0, 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:07.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:07.954 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:07.954 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:07.954 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:07.954 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:07.954 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:09:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f673c0b6680 msgr2=0x7f673c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f673c0b6680 0x7f673c0b8b90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6748031440 tx=0x7f67480058d0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 msgr2=0x7f675819aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 0x7f675819aa20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f675806ca10 tx=0x7f674000ee50 comp rx=0 tx=0).stop 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 shutdown_connections 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6758109e10 0x7f675819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f673c0b6680 0x7f673c0b8b90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67581094c0 0x7f675819aa20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 --2- 172.21.15.39:0/1216657677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6758103b80 0x7f675819a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 >> 172.21.15.39:0/1216657677 conn(0x7f67580ff5d0 msgr2=0x7f67581019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.950+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 shutdown_connections 2024-09-21T01:09:07.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:07.951+0000 7f675da00640 1 -- 172.21.15.39:0/1216657677 wait complete. 2024-09-21T01:09:08.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268260 7.25 MB/sec execute 95 sec latency 8429.481 ms 2024-09-21T01:09:09.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268264 7.17 MB/sec execute 96 sec latency 583.376 ms 2024-09-21T01:09:09.269 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '72706f34', total seconds elapsed: 15.0 2024-09-21T01:09:09.483 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:09 smithi039 ceph-mon[30415]: pgmap v558: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.0 MiB/s wr, 145 op/s 2024-09-21T01:09:09.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:09 smithi150 ceph-mon[34945]: pgmap v558: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.0 MiB/s wr, 145 op/s 2024-09-21T01:09:09.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:09 smithi160 ceph-mon[34696]: pgmap v558: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.0 MiB/s wr, 145 op/s 2024-09-21T01:09:10.148 INFO:tasks.workunit.client.0.smithi039.stdout: 1 268345 7.10 MB/sec execute 97 sec latency 1067.906 ms 2024-09-21T01:09:11.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 269819 7.16 MB/sec execute 98 sec latency 57.516 ms 2024-09-21T01:09:11.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:11 smithi150 ceph-mon[34945]: pgmap v559: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-09-21T01:09:11.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:11 smithi160 ceph-mon[34696]: pgmap v559: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-09-21T01:09:11.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:11 smithi039 ceph-mon[30415]: pgmap v559: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-09-21T01:09:12.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 271270 7.14 MB/sec execute 99 sec latency 17.645 ms 2024-09-21T01:09:13.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273550 7.21 MB/sec execute 100 sec latency 11.564 ms 2024-09-21T01:09:13.455 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:13 smithi039 ceph-mon[30415]: pgmap v560: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.5 MiB/s wr, 87 op/s 2024-09-21T01:09:13.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:13 smithi150 ceph-mon[34945]: pgmap v560: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.5 MiB/s wr, 87 op/s 2024-09-21T01:09:13.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:13 smithi160 ceph-mon[34696]: pgmap v560: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.5 MiB/s wr, 87 op/s 2024-09-21T01:09:14.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273574 7.14 MB/sec execute 101 sec latency 611.613 ms 2024-09-21T01:09:15.144 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 7.07 MB/sec execute 102 sec latency 1040.107 ms 2024-09-21T01:09:15.524 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:15 smithi039 ceph-mon[30415]: pgmap v561: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-09-21T01:09:15.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:15 smithi150 ceph-mon[34945]: pgmap v561: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-09-21T01:09:15.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:15 smithi160 ceph-mon[34696]: pgmap v561: 129 pgs: 129 active+clean; 159 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-09-21T01:09:16.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 7.00 MB/sec execute 103 sec latency 1436.383 ms 2024-09-21T01:09:17.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.93 MB/sec execute 104 sec latency 2436.540 ms 2024-09-21T01:09:17.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:17 smithi150 ceph-mon[34945]: pgmap v562: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.7 MiB/s wr, 118 op/s 2024-09-21T01:09:17.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:17 smithi160 ceph-mon[34696]: pgmap v562: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.7 MiB/s wr, 118 op/s 2024-09-21T01:09:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:17 smithi039 ceph-mon[30415]: pgmap v562: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.7 MiB/s wr, 118 op/s 2024-09-21T01:09:18.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.87 MB/sec execute 105 sec latency 3436.642 ms 2024-09-21T01:09:19.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.80 MB/sec execute 106 sec latency 4436.695 ms 2024-09-21T01:09:19.486 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:19 smithi039 ceph-mon[30415]: pgmap v563: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-09-21T01:09:19.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:19 smithi150 ceph-mon[34945]: pgmap v563: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-09-21T01:09:19.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:19 smithi160 ceph-mon[34696]: pgmap v563: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-09-21T01:09:20.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.74 MB/sec execute 107 sec latency 5436.826 ms 2024-09-21T01:09:21.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.68 MB/sec execute 108 sec latency 6436.904 ms 2024-09-21T01:09:21.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:21 smithi150 ceph-mon[34945]: pgmap v564: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:09:21.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:21 smithi160 ceph-mon[34696]: pgmap v564: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:09:21.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:21 smithi039 ceph-mon[30415]: pgmap v564: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:09:22.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.61 MB/sec execute 109 sec latency 7437.050 ms 2024-09-21T01:09:23.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.55 MB/sec execute 110 sec latency 8437.189 ms 2024-09-21T01:09:23.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:23 smithi039 ceph-mon[30415]: pgmap v565: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.9 MiB/s wr, 125 op/s 2024-09-21T01:09:23.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:23 smithi150 ceph-mon[34945]: pgmap v565: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.9 MiB/s wr, 125 op/s 2024-09-21T01:09:23.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:23 smithi160 ceph-mon[34696]: pgmap v565: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.9 MiB/s wr, 125 op/s 2024-09-21T01:09:24.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.49 MB/sec execute 111 sec latency 9437.261 ms 2024-09-21T01:09:25.145 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.44 MB/sec execute 112 sec latency 10437.356 ms 2024-09-21T01:09:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:25 smithi150 ceph-mon[34945]: pgmap v566: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-09-21T01:09:25.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:25 smithi039 ceph-mon[30415]: pgmap v566: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-09-21T01:09:25.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:25 smithi160 ceph-mon[34696]: pgmap v566: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-09-21T01:09:26.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273747 6.38 MB/sec execute 113 sec latency 11437.456 ms 2024-09-21T01:09:27.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 273893 6.33 MB/sec execute 114 sec latency 12292.445 ms 2024-09-21T01:09:27.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:27 smithi150 ceph-mon[34945]: pgmap v567: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:09:27.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:27 smithi160 ceph-mon[34696]: pgmap v567: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:09:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:27 smithi039 ceph-mon[30415]: pgmap v567: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:09:27.841 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:09:27.841 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-09-21T01:09:27.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:09:28.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 275191 6.32 MB/sec execute 115 sec latency 21.744 ms 2024-09-21T01:09:28.261 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:29.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 276155 6.32 MB/sec execute 116 sec latency 370.827 ms 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 -- 172.21.15.39:0/1346578268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 msgr2=0x7fa330075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 --2- 172.21.15.39:0/1346578268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 0x7fa330075bc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa320009900 tx=0x7fa32002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 -- 172.21.15.39:0/1346578268 shutdown_connections 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 --2- 172.21.15.39:0/1346578268 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa33010ff10 0x7fa330112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 --2- 172.21.15.39:0/1346578268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 --2- 172.21.15.39:0/1346578268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 0x7fa330075bc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 -- 172.21.15.39:0/1346578268 >> 172.21.15.39:0/1346578268 conn(0x7fa3300ff440 msgr2=0x7fa330101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 -- 172.21.15.39:0/1346578268 shutdown_connections 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.202+0000 7fa334c00640 1 -- 172.21.15.39:0/1346578268 wait complete. 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.203+0000 7fa334c00640 1 Processor -- start 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.203+0000 7fa334c00640 1 -- start start 2024-09-21T01:09:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa334c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 0x7fa33010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa334c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa334c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa33010ff10 0x7fa33010d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa334c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa330115b10 con 0x7fa3300757b0 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa334c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa330115990 con 0x7fa330076100 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa334c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa330115c90 con 0x7fa33010ff10 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa32f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 0x7fa33010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa32ea00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa32ea00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:36682/0 (socket says 172.21.15.39:36682) 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa32ea00640 1 -- 172.21.15.39:0/3262753234 learned_addr learned my addr 172.21.15.39:0/3262753234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.204+0000 7fa32fe00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa33010ff10 0x7fa33010d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa32ea00640 1 -- 172.21.15.39:0/3262753234 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa33010ff10 msgr2=0x7fa33010d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa32ea00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa33010ff10 0x7fa33010d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa32ea00640 1 -- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 msgr2=0x7fa33010cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa32ea00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 0x7fa33010cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa32ea00640 1 -- 172.21.15.39:0/3262753234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3301aca00 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa32ea00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010d420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00e9c0 tx=0x7fa31c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c00ccc0 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa31c0035f0 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c00ce60 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.205+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3301accf0 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.206+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3301ad1b0 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.207+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa330075bc0 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.229+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa31c0037a0 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.230+0000 7fa327e00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3140b6680 0x7fa3140b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.231+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa31c0e8210 con 0x7fa330076100 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.231+0000 7fa32f400640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3140b6680 0x7fa3140b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.231+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa31c0e8670 con 0x7fa330076100 2024-09-21T01:09:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.232+0000 7fa32f400640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3140b6680 0x7fa3140b8b90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa320005ab0 tx=0x7fa320005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:29.487 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:29 smithi039 ceph-mon[30415]: pgmap v568: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-21T01:09:29.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:29 smithi150 ceph-mon[34945]: pgmap v568: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-21T01:09:29.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:29 smithi160 ceph-mon[34696]: pgmap v568: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-21T01:09:29.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.639+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3301ad440 con 0x7fa330076100 2024-09-21T01:09:29.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.640+0000 7fa327e00640 1 -- 172.21.15.39:0/3262753234 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11911 (secure 0 0 0) 0x7fa31c0ae380 con 0x7fa330076100 2024-09-21T01:09:29.641 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:09:29.641 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"btime":"2024-09-21T01:08:19:688675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:08:19.501727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:09:29.641 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3140b6680 msgr2=0x7fa3140b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3140b6680 0x7fa3140b8b90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa320005ab0 tx=0x7fa320005a20 comp rx=0 tx=0).stop 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 msgr2=0x7fa33010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010d420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00e9c0 tx=0x7fa31c00edc0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 shutdown_connections 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa33010ff10 0x7fa33010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa3140b6680 0x7fa3140b8b90 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa320005ab0 tx=0x7fa320005a20 comp rx=0 tx=0).stop 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330076100 0x7fa33010d420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 --2- 172.21.15.39:0/3262753234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3300757b0 0x7fa33010cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:29.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.643+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 >> 172.21.15.39:0/3262753234 conn(0x7fa3300ff440 msgr2=0x7fa330109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:29.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.644+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 shutdown_connections 2024-09-21T01:09:29.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:29.644+0000 7fa334c00640 1 -- 172.21.15.39:0/3262753234 wait complete. 2024-09-21T01:09:30.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 276160 6.27 MB/sec execute 117 sec latency 708.027 ms 2024-09-21T01:09:30.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:30 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3262753234' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:30.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:30 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3262753234' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:30.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:30 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3262753234' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:30.913 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0' 2024-09-21T01:09:30.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0 2024-09-21T01:09:31.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 277375 6.27 MB/sec execute 118 sec latency 808.057 ms 2024-09-21T01:09:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:31.513 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:31 smithi039 ceph-mon[30415]: pgmap v569: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 61 op/s 2024-09-21T01:09:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:31 smithi150 ceph-mon[34945]: pgmap v569: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 61 op/s 2024-09-21T01:09:31.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:31 smithi160 ceph-mon[34696]: pgmap v569: 129 pgs: 129 active+clean; 162 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 61 op/s 2024-09-21T01:09:32.146 INFO:tasks.workunit.client.0.smithi039.stdout: 1 278672 6.26 MB/sec execute 119 sec latency 22.995 ms 2024-09-21T01:09:32.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.254+0000 7fa4bfe00640 1 -- 172.21.15.39:0/3587543382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c00757b0 msgr2=0x7fa4c0075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.254+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/3587543382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c00757b0 0x7fa4c0075bc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009900 tx=0x7fa4b002a400 comp rx=0 tx=0).stop 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 -- 172.21.15.39:0/3587543382 shutdown_connections 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/3587543382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c010ff20 0x7fa4c0112360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/3587543382 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c0076100 0x7fa4c010f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/3587543382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c00757b0 0x7fa4c0075bc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 -- 172.21.15.39:0/3587543382 >> 172.21.15.39:0/3587543382 conn(0x7fa4c00ff430 msgr2=0x7fa4c0101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 -- 172.21.15.39:0/3587543382 shutdown_connections 2024-09-21T01:09:32.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.255+0000 7fa4bfe00640 1 -- 172.21.15.39:0/3587543382 wait complete. 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 Processor -- start 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 -- start start 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c00757b0 0x7fa4c019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c0076100 0x7fa4c019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 0x7fa4c01a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0115b20 con 0x7fa4c00757b0 2024-09-21T01:09:32.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01159a0 con 0x7fa4c010ff20 2024-09-21T01:09:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.256+0000 7fa4bfe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0115ca0 con 0x7fa4c0076100 2024-09-21T01:09:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 0x7fa4c01a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 0x7fa4c01a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:36706/0 (socket says 172.21.15.39:36706) 2024-09-21T01:09:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 -- 172.21.15.39:0/4284401324 learned_addr learned my addr 172.21.15.39:0/4284401324 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4be000640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c0076100 0x7fa4c019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bea00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c00757b0 0x7fa4c019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 -- 172.21.15.39:0/4284401324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c0076100 msgr2=0x7fa4c019ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c0076100 0x7fa4c019ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 -- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c00757b0 msgr2=0x7fa4c019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c00757b0 0x7fa4c019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.257+0000 7fa4bf400640 1 -- 172.21.15.39:0/4284401324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c01a2460 con 0x7fa4c010ff20 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.258+0000 7fa4bf400640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 0x7fa4c01a1d30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800efd0 tx=0x7fa4b800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.258+0000 7fa4b7400640 1 -- 172.21.15.39:0/4284401324 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8003b70 con 0x7fa4c010ff20 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.258+0000 7fa4b7400640 1 -- 172.21.15.39:0/4284401324 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b8003d10 con 0x7fa4c010ff20 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.258+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c01a2750 con 0x7fa4c010ff20 2024-09-21T01:09:32.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.258+0000 7fa4b7400640 1 -- 172.21.15.39:0/4284401324 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80108a0 con 0x7fa4c010ff20 2024-09-21T01:09:32.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.258+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c01a8eb0 con 0x7fa4c010ff20 2024-09-21T01:09:32.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.262+0000 7fa4cc287740 1 Processor -- start 2024-09-21T01:09:32.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.263+0000 7fa4b7400640 1 -- 172.21.15.39:0/4284401324 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa4b8004020 con 0x7fa4c010ff20 2024-09-21T01:09:32.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.263+0000 7fa4cc287740 1 -- start start 2024-09-21T01:09:32.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f8c850 0x557cf1333b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf13340d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf1334610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4cc287740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557cf12a9a90 con 0x557cf0f8c850 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4cc287740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557cf1298ec0 con 0x557cf0f6bb70 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4cc287740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557cf129ccd0 con 0x557cf0f63990 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf13340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf13340d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37736/0 (socket says 172.21.15.39:37736) 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4bea00640 1 -- 172.21.15.39:0/3177410428 learned_addr learned my addr 172.21.15.39:0/3177410428 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.264+0000 7fa4be000640 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f8c850 0x557cf1333b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4bea00640 1 -- 172.21.15.39:0/3177410428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 msgr2=0x557cf1334610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:09:32.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4bea00640 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf1334610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4bea00640 1 -- 172.21.15.39:0/3177410428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f8c850 msgr2=0x557cf1333b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:32.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4bea00640 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f8c850 0x557cf1333b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4bea00640 1 -- 172.21.15.39:0/3177410428 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cf123e9d0 con 0x557cf0f63990 2024-09-21T01:09:32.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4bea00640 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf13340d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009230 tx=0x7fa4b0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e48fb9241d4cacfd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:32.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.265+0000 7fa4ab400640 1 -- 172.21.15.39:0/3177410428 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0035040 con 0x557cf0f63990 2024-09-21T01:09:32.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.266+0000 7fa4b7400640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4a40b66f0 0x7fa4a40b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.266+0000 7fa4ab400640 1 -- 172.21.15.39:0/3177410428 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b00054f0 con 0x557cf0f63990 2024-09-21T01:09:32.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.266+0000 7fa4ab400640 1 -- 172.21.15.39:0/3177410428 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00035f0 con 0x557cf0f63990 2024-09-21T01:09:32.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.266+0000 7fa4b7400640 1 -- 172.21.15.39:0/4284401324 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4b80e7e50 con 0x7fa4c010ff20 2024-09-21T01:09:32.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.266+0000 7fa4be000640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4a40b66f0 0x7fa4a40b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 -- 172.21.15.39:0/3177410428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 msgr2=0x557cf13340d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf13340d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009230 tx=0x7fa4b0004040 comp rx=0 tx=0).stop 2024-09-21T01:09:32.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 -- 172.21.15.39:0/3177410428 shutdown_connections 2024-09-21T01:09:32.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf1334610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf13340d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3177410428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f8c850 0x557cf1333b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 -- 172.21.15.39:0/3177410428 >> 172.21.15.39:0/3177410428 conn(0x557cf0f80c40 msgr2=0x557cf1331a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:32.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4be000640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4a40b66f0 0x7fa4a40b8c00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0100390 tx=0x7fa4ac005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:32.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.267+0000 7fa4cc287740 1 -- 172.21.15.39:0/3177410428 shutdown_connections 2024-09-21T01:09:32.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.268+0000 7fa4cc287740 1 -- 172.21.15.39:0/3177410428 wait complete. 2024-09-21T01:09:32.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.268+0000 7fa4cc287740 1 -- start start 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.268+0000 7fa4cc287740 1 Processor -- start 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f63990 0x557cf133f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf133f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf1353e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557cf10fa7b0 con 0x557cf0f63990 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557cf123d650 con 0x557cf0f6bb70 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557cf1342310 con 0x557cf0f8c850 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4cc287740 10 client.0 _ll_register_callbacks cb 0x557cf0fa40a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf1353e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4be000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf133f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf1353e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37752/0 (socket says 172.21.15.39:37752) 2024-09-21T01:09:32.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4be000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf133f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:36724/0 (socket says 172.21.15.39:36724) 2024-09-21T01:09:32.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.269+0000 7fa4bf400640 1 -- 172.21.15.39:0/981338552 learned_addr learned my addr 172.21.15.39:0/981338552 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:32.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bea00640 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f63990 0x557cf133f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bf400640 1 -- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 msgr2=0x557cf133f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bf400640 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf133f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bf400640 1 -- 172.21.15.39:0/981338552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f63990 msgr2=0x557cf133f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bf400640 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f63990 0x557cf133f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bf400640 1 -- 172.21.15.39:0/981338552 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cf1006ba0 con 0x557cf0f8c850 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa4bf400640 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf1353e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80ad620 tx=0x7fa4b8004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b18ca3374a2f57db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa49fe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:09:32.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.270+0000 7fa49fe00640 1 -- 172.21.15.39:0/981338552 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80ad430 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa49fe00640 1 -- 172.21.15.39:0/981338552 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4b8004b40 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4b00054f0 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa49fe00640 1 -- 172.21.15.39:0/981338552 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa4b80ae020 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 10 client.25747 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557cf13569e0 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa49fe00640 1 -- 172.21.15.39:0/981338552 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b00054f0 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa49fe00640 1 -- 172.21.15.39:0/981338552 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fa4b801f020 con 0x557cf0f8c850 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 10 client.25747 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 10 client.25747 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 20 client.25747 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.271+0000 7fa4cc287740 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f4b1d0 0x557cf1359ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.272+0000 7fa4cc287740 4 client.25747 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.272+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cf12ab9e0 con 0x557cf0f4b1d0 2024-09-21T01:09:32.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.272+0000 7fa4bea00640 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f4b1d0 0x557cf1359ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.272+0000 7fa49fe00640 10 client.25747 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:32.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.272+0000 7fa4bea00640 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f4b1d0 0x557cf1359ef0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3389695eb6a4caa0 server_cookie=a5e91453bac95202 in_seq=0 out_seq=0 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.274+0000 7fa49fe00640 1 -- 172.21.15.39:0/981338552 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557cf12ab9e0 con 0x557cf0f4b1d0 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.274+0000 7fa49fe00640 10 client.25747 handle_command_reply: tid=0 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.274+0000 7fa4cc287740 1 client.25747 shutdown 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.274+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 msgr2=0x557cf1353e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.274+0000 7fa4cc287740 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf1353e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80ad620 tx=0x7fa4b8004790 comp rx=0 tx=0).stop 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 shutdown_connections 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f4b1d0 0x557cf1359ef0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf1353e40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f6bb70 0x557cf133f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 --2- 172.21.15.39:0/981338552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f63990 0x557cf133f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa49fe00640 0 client.25747 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:32.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 >> 172.21.15.39:0/981338552 conn(0x557cf0f80c40 msgr2=0x557cf1336d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:32.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 shutdown_connections 2024-09-21T01:09:32.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 1 -- 172.21.15.39:0/981338552 wait complete. 2024-09-21T01:09:32.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.275+0000 7fa4cc287740 20 client.25747 trim_cache size 0 max 16384 2024-09-21T01:09:32.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 Processor -- start 2024-09-21T01:09:32.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 -- start start 2024-09-21T01:09:32.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf1330a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf13352d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f8c850 0x557cf133c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01b1390 con 0x557cf0f6bb70 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.306+0000 7fa4cc287740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01aec60 con 0x557cf0f8c850 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4cc287740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01acaf0 con 0x557cf0f63990 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf1330a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf1330a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37760/0 (socket says 172.21.15.39:37760) 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bea00640 1 -- 172.21.15.39:0/3972093760 learned_addr learned my addr 172.21.15.39:0/3972093760 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bf400640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f8c850 0x557cf133c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4be000640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf13352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bf400640 1 -- 172.21.15.39:0/3972093760 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 msgr2=0x557cf1330a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bf400640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf1330a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bf400640 1 -- 172.21.15.39:0/3972093760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 msgr2=0x557cf13352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:32.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bf400640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf13352d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bf400640 1 -- 172.21.15.39:0/3972093760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cf133f770 con 0x557cf0f8c850 2024-09-21T01:09:32.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4bea00640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf1330a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:09:32.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.307+0000 7fa4be000640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf13352d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:09:32.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.308+0000 7fa4bf400640 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f8c850 0x557cf133c610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80aab90 tx=0x7fa4b80b22e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b525773c88db33fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:32.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.308+0000 7fa4aaa00640 1 -- 172.21.15.39:0/3972093760 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80a9090 con 0x557cf0f8c850 2024-09-21T01:09:32.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.308+0000 7fa4aaa00640 1 -- 172.21.15.39:0/3972093760 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b80ad430 con 0x557cf0f8c850 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.308+0000 7fa4aaa00640 1 -- 172.21.15.39:0/3972093760 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8004da0 con 0x557cf0f8c850 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 -- 172.21.15.39:0/3972093760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f8c850 msgr2=0x557cf133c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f8c850 0x557cf133c610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80aab90 tx=0x7fa4b80b22e0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 -- 172.21.15.39:0/3972093760 shutdown_connections 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f8c850 0x557cf133c610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf13352d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 --2- 172.21.15.39:0/3972093760 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f63990 0x557cf1330a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 -- 172.21.15.39:0/3972093760 >> 172.21.15.39:0/3972093760 conn(0x557cf0f80c40 msgr2=0x557cf1334bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 -- 172.21.15.39:0/3972093760 shutdown_connections 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.309+0000 7fa4cc287740 1 -- 172.21.15.39:0/3972093760 wait complete. 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.310+0000 7fa4cc287740 1 -- start start 2024-09-21T01:09:32.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.310+0000 7fa4cc287740 1 Processor -- start 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 0x557cf1350690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf134ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf134b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01ad140 con 0x557cf0f6bb70 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01ae590 con 0x557cf0f63990 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01ad7f0 con 0x557cf0f8c850 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4cc287740 10 client.0 _ll_register_callbacks cb 0x7fa4b8003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 0x557cf1350690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 0x557cf1350690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:36746/0 (socket says 172.21.15.39:36746) 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4bea00640 1 -- 172.21.15.39:0/2248125509 learned_addr learned my addr 172.21.15.39:0/2248125509 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.311+0000 7fa4bf400640 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf134b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa4bea00640 1 -- 172.21.15.39:0/2248125509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 msgr2=0x557cf134b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa4bea00640 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf134b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa4bea00640 1 -- 172.21.15.39:0/2248125509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 msgr2=0x557cf134ade0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa4bea00640 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf134ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa4bea00640 1 -- 172.21.15.39:0/2248125509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cf134baf0 con 0x557cf0f63990 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa4bea00640 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 0x557cf1350690 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0005ab0 tx=0x7fa4b002a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a07170cd6d863405 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.312+0000 7fa49fe00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.313+0000 7fa49fe00640 1 -- 172.21.15.39:0/2248125509 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b005c060 con 0x557cf0f63990 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.313+0000 7fa49fe00640 1 -- 172.21.15.39:0/2248125509 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4b00037f0 con 0x557cf0f63990 2024-09-21T01:09:32.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.313+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4b80ad430 con 0x557cf0f63990 2024-09-21T01:09:32.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa49fe00640 1 -- 172.21.15.39:0/2248125509 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa4b0035040 con 0x557cf0f63990 2024-09-21T01:09:32.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 10 client.25763 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:32.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557cf13c5810 con 0x557cf0f63990 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa49fe00640 1 -- 172.21.15.39:0/2248125509 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b80ad430 con 0x557cf0f63990 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa49fe00640 1 -- 172.21.15.39:0/2248125509 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fa4b0005070 con 0x557cf0f63990 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 10 client.25763 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 10 client.25763 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 20 client.25763 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f20580 0x557cf13c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 4 client.25763 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}] 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.314+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}) -- 0x557cf12a7490 con 0x557cf0f20580 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.315+0000 7fa4be000640 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f20580 0x557cf13c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.315+0000 7fa4be000640 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f20580 0x557cf13c7b00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89a8b5e89fdce774 server_cookie=54266ab0fbb4c714 in_seq=0 out_seq=0 2024-09-21T01:09:32.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.315+0000 7fa49fe00640 10 client.25763 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:32.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.524+0000 7fa49fe00640 1 -- 172.21.15.39:0/2248125509 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557cf12a7490 con 0x557cf0f20580 2024-09-21T01:09:32.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.524+0000 7fa49fe00640 10 client.25763 handle_command_reply: tid=0 2024-09-21T01:09:32.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.524+0000 7fa4cc287740 1 client.25763 shutdown 2024-09-21T01:09:32.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 msgr2=0x557cf1350690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 0x557cf1350690 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0005ab0 tx=0x7fa4b002a400 comp rx=0 tx=0).stop 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 shutdown_connections 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cf0f20580 0x557cf13c7b00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cf0f8c850 0x557cf134b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cf0f6bb70 0x557cf134ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa49fe00640 0 client.25763 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 --2- 172.21.15.39:0/2248125509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cf0f63990 0x557cf1350690 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 >> 172.21.15.39:0/2248125509 conn(0x557cf0f80c40 msgr2=0x557cf13408d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 shutdown_connections 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 1 -- 172.21.15.39:0/2248125509 wait complete. 2024-09-21T01:09:32.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.525+0000 7fa4cc287740 20 client.25763 trim_cache size 0 max 16384 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 30, 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 22, 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "1c5790c3": { 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 22, 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:09:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.2, 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:32.528 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4a40b66f0 msgr2=0x7fa4a40b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4a40b66f0 0x7fa4a40b8c00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0100390 tx=0x7fa4ac005f90 comp rx=0 tx=0).stop 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 msgr2=0x7fa4c01a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 0x7fa4c01a1d30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800efd0 tx=0x7fa4b800c6a0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 shutdown_connections 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa4a40b66f0 0x7fa4a40b8c00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c010ff20 0x7fa4c01a1d30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa4c0076100 0x7fa4c019ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 --2- 172.21.15.39:0/4284401324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c00757b0 0x7fa4c019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:32.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 >> 172.21.15.39:0/4284401324 conn(0x7fa4c00ff430 msgr2=0x7fa4c0109ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:32.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 shutdown_connections 2024-09-21T01:09:32.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:32.526+0000 7fa4bfe00640 1 -- 172.21.15.39:0/4284401324 wait complete. 2024-09-21T01:09:33.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 6.23 MB/sec execute 120 sec latency 801.657 ms 2024-09-21T01:09:33.364 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1c5790c3', quiesce took 5.5 seconds. Will release after: 13.75 2024-09-21T01:09:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:33 smithi150 ceph-mon[34945]: pgmap v570: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:09:33.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:33 smithi160 ceph-mon[34696]: pgmap v570: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:09:33.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:33 smithi039 ceph-mon[30415]: pgmap v570: 129 pgs: 129 active+clean; 163 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:09:34.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 6.18 MB/sec execute 121 sec latency 1801.848 ms 2024-09-21T01:09:35.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 6.13 MB/sec execute 122 sec latency 2802.005 ms 2024-09-21T01:09:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:35 smithi160 ceph-mon[34696]: pgmap v571: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-09-21T01:09:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:35 smithi039 ceph-mon[30415]: pgmap v571: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-09-21T01:09:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:35 smithi150 ceph-mon[34945]: pgmap v571: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-09-21T01:09:36.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 6.08 MB/sec execute 123 sec latency 3802.170 ms 2024-09-21T01:09:37.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 6.03 MB/sec execute 124 sec latency 4802.311 ms 2024-09-21T01:09:37.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:37 smithi160 ceph-mon[34696]: pgmap v572: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.5 MiB/s wr, 106 op/s 2024-09-21T01:09:37.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:37 smithi039 ceph-mon[30415]: pgmap v572: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.5 MiB/s wr, 106 op/s 2024-09-21T01:09:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:37 smithi150 ceph-mon[34945]: pgmap v572: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.5 MiB/s wr, 106 op/s 2024-09-21T01:09:38.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.98 MB/sec execute 125 sec latency 5802.453 ms 2024-09-21T01:09:39.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.93 MB/sec execute 126 sec latency 6802.601 ms 2024-09-21T01:09:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:39 smithi039 ceph-mon[30415]: pgmap v573: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-09-21T01:09:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:39 smithi150 ceph-mon[34945]: pgmap v573: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-09-21T01:09:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:39 smithi160 ceph-mon[34696]: pgmap v573: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-09-21T01:09:40.147 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.89 MB/sec execute 127 sec latency 7802.683 ms 2024-09-21T01:09:41.148 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.84 MB/sec execute 128 sec latency 8802.790 ms 2024-09-21T01:09:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:41 smithi039 ceph-mon[30415]: pgmap v574: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-09-21T01:09:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:41 smithi150 ceph-mon[34945]: pgmap v574: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-09-21T01:09:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:41 smithi160 ceph-mon[34696]: pgmap v574: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-09-21T01:09:42.148 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.80 MB/sec execute 129 sec latency 9802.929 ms 2024-09-21T01:09:43.148 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.75 MB/sec execute 130 sec latency 10803.027 ms 2024-09-21T01:09:43.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:43 smithi039 ceph-mon[30415]: pgmap v575: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 858 KiB/s wr, 82 op/s 2024-09-21T01:09:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:43 smithi150 ceph-mon[34945]: pgmap v575: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 858 KiB/s wr, 82 op/s 2024-09-21T01:09:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:43 smithi160 ceph-mon[34696]: pgmap v575: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 858 KiB/s wr, 82 op/s 2024-09-21T01:09:44.156 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.71 MB/sec execute 131 sec latency 11811.603 ms 2024-09-21T01:09:45.156 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.66 MB/sec execute 132 sec latency 12811.665 ms 2024-09-21T01:09:45.506 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:45 smithi039 ceph-mon[30415]: pgmap v576: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 501 KiB/s wr, 60 op/s 2024-09-21T01:09:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:45 smithi150 ceph-mon[34945]: pgmap v576: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 501 KiB/s wr, 60 op/s 2024-09-21T01:09:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:45 smithi160 ceph-mon[34696]: pgmap v576: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 501 KiB/s wr, 60 op/s 2024-09-21T01:09:46.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.62 MB/sec execute 133 sec latency 13811.727 ms 2024-09-21T01:09:47.115 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:09:47.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.58 MB/sec execute 134 sec latency 14811.786 ms 2024-09-21T01:09:47.498 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:47.637 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:47 smithi039 ceph-mon[30415]: pgmap v577: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 418 KiB/s wr, 43 op/s 2024-09-21T01:09:47.658 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:47 smithi160 ceph-mon[34696]: pgmap v577: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 418 KiB/s wr, 43 op/s 2024-09-21T01:09:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:47 smithi150 ceph-mon[34945]: pgmap v577: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 418 KiB/s wr, 43 op/s 2024-09-21T01:09:48.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.54 MB/sec execute 135 sec latency 15811.847 ms 2024-09-21T01:09:48.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 -- 172.21.15.39:0/2668359279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010b6c0 msgr2=0x7ff0a010bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:48.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2668359279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010b6c0 0x7ff0a010bad0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff088009900 tx=0x7ff08802a410 comp rx=0 tx=0).stop 2024-09-21T01:09:48.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 -- 172.21.15.39:0/2668359279 shutdown_connections 2024-09-21T01:09:48.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2668359279 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a010c010 0x7ff0a0112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2668359279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010b6c0 0x7ff0a010bad0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2668359279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a0105d80 0x7ff0a0106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 -- 172.21.15.39:0/2668359279 >> 172.21.15.39:0/2668359279 conn(0x7ff0a0101810 msgr2=0x7ff0a0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:48.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 -- 172.21.15.39:0/2668359279 shutdown_connections 2024-09-21T01:09:48.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.174+0000 7ff0a5400640 1 -- 172.21.15.39:0/2668359279 wait complete. 2024-09-21T01:09:48.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.175+0000 7ff0a5400640 1 Processor -- start 2024-09-21T01:09:48.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.175+0000 7ff0a5400640 1 -- start start 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.175+0000 7ff0a5400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 0x7ff0a019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.175+0000 7ff0a5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a010b6c0 0x7ff0a019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.175+0000 7ff0a5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010c010 0x7ff0a01a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff0a5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0115d20 con 0x7ff0a010c010 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff0a5400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0115ba0 con 0x7ff0a010b6c0 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff0a5400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0115ea0 con 0x7ff0a0105d80 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 0x7ff0a019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 0x7ff0a019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59252/0 (socket says 172.21.15.39:59252) 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010c010 0x7ff0a01a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:48.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 -- 172.21.15.39:0/2592591786 learned_addr learned my addr 172.21.15.39:0/2592591786 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:48.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff097e00640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a010b6c0 0x7ff0a019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:48.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 -- 172.21.15.39:0/2592591786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a010b6c0 msgr2=0x7ff0a019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:48.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a010b6c0 0x7ff0a019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 -- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010c010 msgr2=0x7ff0a01a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:48.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010c010 0x7ff0a01a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.176+0000 7ff09e800640 1 -- 172.21.15.39:0/2592591786 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a01a2640 con 0x7ff0a0105d80 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.177+0000 7ff09e800640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 0x7ff0a019c6d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff09000e9f0 tx=0x7ff09000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.177+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09000cd80 con 0x7ff0a0105d80 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.177+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a01a2930 con 0x7ff0a0105d80 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.177+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0900108e0 con 0x7ff0a0105d80 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.177+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090018980 con 0x7ff0a0105d80 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.177+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a01a9090 con 0x7ff0a0105d80 2024-09-21T01:09:48.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.178+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a0106190 con 0x7ff0a0105d80 2024-09-21T01:09:48.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.181+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff090010430 con 0x7ff0a0105d80 2024-09-21T01:09:48.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.182+0000 7ff097400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0840b6680 0x7ff0840b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:48.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.183+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff0900e82b0 con 0x7ff0a0105d80 2024-09-21T01:09:48.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.183+0000 7ff097e00640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0840b6680 0x7ff0840b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:48.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.183+0000 7ff097e00640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0840b6680 0x7ff0840b8b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff08802aa00 tx=0x7ff088005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:48.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.184+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff0900b2800 con 0x7ff0a0105d80 2024-09-21T01:09:48.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.421+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0a019d800 con 0x7ff0a0105d80 2024-09-21T01:09:48.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.423+0000 7ff097400640 1 -- 172.21.15.39:0/2592591786 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11911 (secure 0 0 0) 0x7ff0900b7740 con 0x7ff0a0105d80 2024-09-21T01:09:48.424 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:09:48.424 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"btime":"2024-09-21T01:08:19:688675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:08:19.501727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:09:48.425 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-09-21T01:09:48.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0840b6680 msgr2=0x7ff0840b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0840b6680 0x7ff0840b8b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff08802aa00 tx=0x7ff088005a20 comp rx=0 tx=0).stop 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 msgr2=0x7ff0a019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 0x7ff0a019c6d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff09000e9f0 tx=0x7ff09000eeb0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 shutdown_connections 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0a010c010 0x7ff0a01a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0840b6680 0x7ff0840b8b90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a010b6c0 0x7ff0a019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 --2- 172.21.15.39:0/2592591786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0a0105d80 0x7ff0a019c6d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 >> 172.21.15.39:0/2592591786 conn(0x7ff0a0101810 msgr2=0x7ff0a010ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.425+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 shutdown_connections 2024-09-21T01:09:48.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:48.426+0000 7ff0a5400640 1 -- 172.21.15.39:0/2592591786 wait complete. 2024-09-21T01:09:49.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.50 MB/sec execute 136 sec latency 16811.929 ms 2024-09-21T01:09:49.517 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 1c5790c3 --release --await' 2024-09-21T01:09:49.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 1c5790c3 --release --await 2024-09-21T01:09:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:49 smithi039 ceph-mon[30415]: pgmap v578: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 580 KiB/s wr, 25 op/s 2024-09-21T01:09:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:49 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2592591786' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:49 smithi150 ceph-mon[34945]: pgmap v578: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 580 KiB/s wr, 25 op/s 2024-09-21T01:09:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:49 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2592591786' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:49 smithi160 ceph-mon[34696]: pgmap v578: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 580 KiB/s wr, 25 op/s 2024-09-21T01:09:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:49 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2592591786' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:49.939 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:50.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 279173 5.46 MB/sec execute 137 sec latency 17812.051 ms 2024-09-21T01:09:50.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 -- 172.21.15.39:0/1372087159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb441094c0 msgr2=0x7fbb441098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/1372087159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb441094c0 0x7fbb441098d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009900 tx=0x7fbb2c02a410 comp rx=0 tx=0).stop 2024-09-21T01:09:50.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 -- 172.21.15.39:0/1372087159 shutdown_connections 2024-09-21T01:09:50.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/1372087159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb44109e10 0x7fbb44110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/1372087159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb441094c0 0x7fbb441098d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/1372087159 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb44103b80 0x7fbb44103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 -- 172.21.15.39:0/1372087159 >> 172.21.15.39:0/1372087159 conn(0x7fbb440ff610 msgr2=0x7fbb44101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:50.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 -- 172.21.15.39:0/1372087159 shutdown_connections 2024-09-21T01:09:50.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.584+0000 7fbb4ae00640 1 -- 172.21.15.39:0/1372087159 wait complete. 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.585+0000 7fbb4ae00640 1 Processor -- start 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.585+0000 7fbb4ae00640 1 -- start start 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb4ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb44103b80 0x7fbb4419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb4ae00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb441094c0 0x7fbb4419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb4ae00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb44109e10 0x7fbb441a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb4ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbb44113ad0 con 0x7fbb44103b80 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb4ae00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb44113950 con 0x7fbb441094c0 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb4ae00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbb44113c50 con 0x7fbb44109e10 2024-09-21T01:09:50.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb44103b80 0x7fbb4419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb44103b80 0x7fbb4419c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49320/0 (socket says 172.21.15.39:49320) 2024-09-21T01:09:50.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb43e00640 1 -- 172.21.15.39:0/265444675 learned_addr learned my addr 172.21.15.39:0/265444675 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:50.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb3b400640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb441094c0 0x7fbb4419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb48e00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb44109e10 0x7fbb441a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb3b400640 1 -- 172.21.15.39:0/265444675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb44109e10 msgr2=0x7fbb441a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb3b400640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb44109e10 0x7fbb441a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb3b400640 1 -- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb44103b80 msgr2=0x7fbb4419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb3b400640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb44103b80 0x7fbb4419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.586+0000 7fbb3b400640 1 -- 172.21.15.39:0/265444675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb441a2630 con 0x7fbb441094c0 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.587+0000 7fbb3b400640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb441094c0 0x7fbb4419cc60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c02a930 tx=0x7fbb2c031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.587+0000 7fbb41600640 1 -- 172.21.15.39:0/265444675 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c03f070 con 0x7fbb441094c0 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.587+0000 7fbb41600640 1 -- 172.21.15.39:0/265444675 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb2c003aa0 con 0x7fbb441094c0 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.587+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb441a28c0 con 0x7fbb441094c0 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.587+0000 7fbb41600640 1 -- 172.21.15.39:0/265444675 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c035070 con 0x7fbb441094c0 2024-09-21T01:09:50.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.587+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb441a9080 con 0x7fbb441094c0 2024-09-21T01:09:50.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 Processor -- start 2024-09-21T01:09:50.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 -- start start 2024-09-21T01:09:50.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb41600640 1 -- 172.21.15.39:0/265444675 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fbb2c003c50 con 0x7fbb441094c0 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 0x560cbf2ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbeede660 0x560cbf2ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560cbf225290 con 0x560cbeede660 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560cbf2146c0 con 0x560cbeee6840 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.591+0000 7fbb51a33740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560cbf2184d0 con 0x560cbef07520 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 0x560cbf2ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbeede660 0x560cbf2ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 0x560cbf2ae1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59276/0 (socket says 172.21.15.39:59276) 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 -- 172.21.15.39:0/3700505012 learned_addr learned my addr 172.21.15.39:0/3700505012 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:50.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb3b400640 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 -- 172.21.15.39:0/3700505012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 msgr2=0x560cbf2aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 -- 172.21.15.39:0/3700505012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbeede660 msgr2=0x560cbf2ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbeede660 0x560cbf2ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 -- 172.21.15.39:0/3700505012 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cbf1a1460 con 0x560cbef07520 2024-09-21T01:09:50.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb43e00640 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbeede660 0x560cbf2ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.592+0000 7fbb48e00640 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 0x560cbf2ae1c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c011440 tx=0x7fbb3c011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48156c4d4c9e05cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.593+0000 7fbb39600640 1 -- 172.21.15.39:0/3700505012 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c003c30 con 0x560cbef07520 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.593+0000 7fbb39600640 1 -- 172.21.15.39:0/3700505012 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb3c003dd0 con 0x560cbef07520 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.593+0000 7fbb39600640 1 -- 172.21.15.39:0/3700505012 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00fa80 con 0x560cbef07520 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.593+0000 7fbb41600640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbb280b6680 0x7fbb280b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.593+0000 7fbb41600640 1 -- 172.21.15.39:0/265444675 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbb2c107bb0 con 0x7fbb441094c0 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb43e00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbb280b6680 0x7fbb280b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 -- 172.21.15.39:0/3700505012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 msgr2=0x560cbf2ae1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 0x560cbf2ae1c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c011440 tx=0x7fbb3c011900 comp rx=0 tx=0).stop 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 -- 172.21.15.39:0/3700505012 shutdown_connections 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbeede660 0x560cbf2ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3700505012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbef07520 0x560cbf2ae1c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 -- 172.21.15.39:0/3700505012 >> 172.21.15.39:0/3700505012 conn(0x560cbeefb910 msgr2=0x560cbf2ac0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:50.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 -- 172.21.15.39:0/3700505012 shutdown_connections 2024-09-21T01:09:50.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb51a33740 1 -- 172.21.15.39:0/3700505012 wait complete. 2024-09-21T01:09:50.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.594+0000 7fbb43e00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbb280b6680 0x7fbb280b8b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x560cbf2ace40 tx=0x7fbb34005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:50.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.595+0000 7fbb51a33740 1 -- start start 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.595+0000 7fbb51a33740 1 Processor -- start 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2b9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2ba390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560cbf075bb0 con 0x560cbef07520 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560cbf1af6c0 con 0x560cbeede660 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x560cbf2bc9a0 con 0x560cbeee6840 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb51a33740 10 client.0 _ll_register_callbacks cb 0x560cbf1ca490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2ba390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2ba390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59290/0 (socket says 172.21.15.39:59290) 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 -- 172.21.15.39:0/3981808377 learned_addr learned my addr 172.21.15.39:0/3981808377 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb43e00640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2b9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb48e00640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 -- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 msgr2=0x560cbf2b9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2b9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 -- 172.21.15.39:0/3981808377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 msgr2=0x560cbf2c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.596+0000 7fbb3b400640 1 -- 172.21.15.39:0/3981808377 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cbef72b00 con 0x560cbeee6840 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb3b400640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2ba390 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0ca890 tx=0x7fbb2c03b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b626f266154f0b84 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb32000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb32000640 1 -- 172.21.15.39:0/3981808377 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c048070 con 0x560cbeee6840 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb32000640 1 -- 172.21.15.39:0/3981808377 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbb2c0d6db0 con 0x560cbeee6840 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb3c003dd0 con 0x560cbeee6840 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb32000640 1 -- 172.21.15.39:0/3981808377 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbb2c0cd3f0 con 0x560cbeee6840 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb51a33740 10 client.25771 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:50.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.597+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560cbefe9960 con 0x560cbeee6840 2024-09-21T01:09:50.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb32000640 1 -- 172.21.15.39:0/3981808377 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb3c003dd0 con 0x560cbeee6840 2024-09-21T01:09:50.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb32000640 1 -- 172.21.15.39:0/3981808377 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fbb2c04b060 con 0x560cbeee6840 2024-09-21T01:09:50.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb51a33740 10 client.25771 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:50.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb51a33740 10 client.25771 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:50.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb51a33740 20 client.25771 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbeec5ea0 0x560cbf2cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb51a33740 4 client.25771 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.598+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cbf2271e0 con 0x560cbeec5ea0 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.599+0000 7fbb43e00640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbeec5ea0 0x560cbf2cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.599+0000 7fbb43e00640 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbeec5ea0 0x560cbf2cbc10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=685784f07255324b server_cookie=49cb464dd9bd8ed7 in_seq=0 out_seq=0 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.599+0000 7fbb32000640 10 client.25771 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb32000640 1 -- 172.21.15.39:0/3981808377 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560cbf2271e0 con 0x560cbeec5ea0 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb32000640 10 client.25771 handle_command_reply: tid=0 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 client.25771 shutdown 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 msgr2=0x560cbf2ba390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2ba390 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0ca890 tx=0x7fbb2c03b690 comp rx=0 tx=0).stop 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 shutdown_connections 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbeec5ea0 0x560cbf2cbc10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2ba390 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb32000640 0 client.25771 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 --2- 172.21.15.39:0/3981808377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2b9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.600+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 >> 172.21.15.39:0/3981808377 conn(0x560cbeefb910 msgr2=0x560cbf2b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.601+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 shutdown_connections 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.601+0000 7fbb51a33740 1 -- 172.21.15.39:0/3981808377 wait complete. 2024-09-21T01:09:50.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.601+0000 7fbb51a33740 20 client.25771 trim_cache size 0 max 16384 2024-09-21T01:09:50.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.619+0000 7fbb51a33740 1 Processor -- start 2024-09-21T01:09:50.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.619+0000 7fbb51a33740 1 -- start start 2024-09-21T01:09:50.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2bb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb51a33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbb441b58e0 con 0x560cbef07520 2024-09-21T01:09:50.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb51a33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb441b31b0 con 0x560cbeede660 2024-09-21T01:09:50.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb51a33740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbb441b1000 con 0x560cbeee6840 2024-09-21T01:09:50.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49334/0 (socket says 172.21.15.39:49334) 2024-09-21T01:09:50.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 -- 172.21.15.39:0/1191216630 learned_addr learned my addr 172.21.15.39:0/1191216630 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:50.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb3b400640 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb43e00640 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 -- 172.21.15.39:0/1191216630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 msgr2=0x560cbf2bb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2bb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 -- 172.21.15.39:0/1191216630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 msgr2=0x560cbf2c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb48e00640 1 -- 172.21.15.39:0/1191216630 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cbf2aa3b0 con 0x560cbef07520 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.620+0000 7fbb43e00640 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.621+0000 7fbb48e00640 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2b6de0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c009d80 tx=0x7fbb3c009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3aa9e149e39eb17 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.621+0000 7fbb38c00640 1 -- 172.21.15.39:0/1191216630 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c003cb0 con 0x560cbef07520 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.621+0000 7fbb38c00640 1 -- 172.21.15.39:0/1191216630 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb3c00f9f0 con 0x560cbef07520 2024-09-21T01:09:50.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.621+0000 7fbb38c00640 1 -- 172.21.15.39:0/1191216630 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c005540 con 0x560cbef07520 2024-09-21T01:09:50.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 -- 172.21.15.39:0/1191216630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 msgr2=0x560cbf2b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2b6de0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c009d80 tx=0x7fbb3c009db0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 -- 172.21.15.39:0/1191216630 shutdown_connections 2024-09-21T01:09:50.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2b6de0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeee6840 0x560cbf2bb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1191216630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeede660 0x560cbf2c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 -- 172.21.15.39:0/1191216630 >> 172.21.15.39:0/1191216630 conn(0x560cbeefb910 msgr2=0x560cbf2bb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 -- 172.21.15.39:0/1191216630 shutdown_connections 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.622+0000 7fbb51a33740 1 -- 172.21.15.39:0/1191216630 wait complete. 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 -- start start 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 Processor -- start 2024-09-21T01:09:50.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 0x560cbf2c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2be7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2bed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbb441b1670 con 0x560cbef07520 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb441b2ae0 con 0x560cbeee6840 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.623+0000 7fbb51a33740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbb441b1d40 con 0x560cbeede660 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb51a33740 10 client.0 _ll_register_callbacks cb 0x7fbb2c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb48e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2bed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 0x560cbf2c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 0x560cbf2c2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59310/0 (socket says 172.21.15.39:59310) 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb48e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2bed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49342/0 (socket says 172.21.15.39:49342) 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 -- 172.21.15.39:0/1534986502 learned_addr learned my addr 172.21.15.39:0/1534986502 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb3b400640 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2be7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 -- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 msgr2=0x560cbf2be7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2be7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 -- 172.21.15.39:0/1534986502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 msgr2=0x560cbf2bed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:50.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2bed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.624+0000 7fbb43e00640 1 -- 172.21.15.39:0/1534986502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cbf2bf420 con 0x560cbeede660 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb43e00640 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 0x560cbf2c2330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb3401e560 tx=0x7fbb340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbda729215c9ea53 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb32000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb32000640 1 -- 172.21.15.39:0/1534986502 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34035060 con 0x560cbeede660 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb3c00f9f0 con 0x560cbeede660 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb32000640 1 -- 172.21.15.39:0/1534986502 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbb34018030 con 0x560cbeede660 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb32000640 1 -- 172.21.15.39:0/1534986502 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbb3403c050 con 0x560cbeede660 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb51a33740 10 client.25777 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.625+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560cbf337550 con 0x560cbeede660 2024-09-21T01:09:50.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb32000640 1 -- 172.21.15.39:0/1534986502 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb3c00f9f0 con 0x560cbeede660 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb32000640 1 -- 172.21.15.39:0/1534986502 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fbb3402a5e0 con 0x560cbeede660 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb51a33740 10 client.25777 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb51a33740 10 client.25777 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb51a33740 20 client.25777 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbee9b250 0x560cbf339cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb51a33740 4 client.25777 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1c5790c3", "release": true, "await": true}] 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1c5790c3", "release": true, "await": true}) -- 0x560cbf222c90 con 0x560cbee9b250 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.626+0000 7fbb3b400640 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbee9b250 0x560cbf339cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:50.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.627+0000 7fbb3b400640 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbee9b250 0x560cbf339cf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50d23cfaf1fcf50 server_cookie=5b7eb2ce4820bfa4 in_seq=0 out_seq=0 2024-09-21T01:09:50.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.627+0000 7fbb32000640 10 client.25777 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb32000640 1 -- 172.21.15.39:0/1534986502 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560cbf222c90 con 0x560cbee9b250 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb32000640 10 client.25777 handle_command_reply: tid=0 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 client.25777 shutdown 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 msgr2=0x560cbf2c2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 0x560cbf2c2330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb3401e560 tx=0x7fbb340046c0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 shutdown_connections 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560cbee9b250 0x560cbf339cf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560cbef07520 0x560cbf2bed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560cbeee6840 0x560cbf2be7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb32000640 0 client.25777 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 --2- 172.21.15.39:0/1534986502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560cbeede660 0x560cbf2c2330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.634+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 >> 172.21.15.39:0/1534986502 conn(0x560cbeefb910 msgr2=0x560cbf2b8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:50.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 shutdown_connections 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb51a33740 1 -- 172.21.15.39:0/1534986502 wait complete. 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb51a33740 20 client.25777 trim_cache size 0 max 16384 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 30, 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 24, 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "1c5790c3": { 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 24, 2024-09-21T01:09:50.637 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.2, 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:50.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbb280b6680 msgr2=0x7fbb280b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbb280b6680 0x7fbb280b8b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x560cbf2ace40 tx=0x7fbb34005f90 comp rx=0 tx=0).stop 2024-09-21T01:09:50.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb441094c0 msgr2=0x7fbb4419cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.635+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb441094c0 0x7fbb4419cc60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c02a930 tx=0x7fbb2c031a20 comp rx=0 tx=0).stop 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 shutdown_connections 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb44109e10 0x7fbb441a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fbb280b6680 0x7fbb280b8b90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb441094c0 0x7fbb4419cc60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 --2- 172.21.15.39:0/265444675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb44103b80 0x7fbb4419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 >> 172.21.15.39:0/265444675 conn(0x7fbb440ff610 msgr2=0x7fbb44106b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 shutdown_connections 2024-09-21T01:09:50.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:50.636+0000 7fbb4ae00640 1 -- 172.21.15.39:0/265444675 wait complete. 2024-09-21T01:09:51.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 280356 5.49 MB/sec execute 138 sec latency 18289.588 ms 2024-09-21T01:09:51.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:51 smithi039 ceph-mon[30415]: pgmap v579: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 593 KiB/s wr, 25 op/s 2024-09-21T01:09:51.772 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1c5790c3', total seconds elapsed: 23.9 2024-09-21T01:09:51.772 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-09-21T01:09:51.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:09:51.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:51 smithi150 ceph-mon[34945]: pgmap v579: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 593 KiB/s wr, 25 op/s 2024-09-21T01:09:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:51 smithi160 ceph-mon[34696]: pgmap v579: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 593 KiB/s wr, 25 op/s 2024-09-21T01:09:52.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 281807 5.50 MB/sec execute 139 sec latency 16.578 ms 2024-09-21T01:09:52.208 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:53.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.054+0000 7f2d20a00640 1 -- 172.21.15.39:0/4168942884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 msgr2=0x7f2d1c112330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:53.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.054+0000 7f2d20a00640 1 --2- 172.21.15.39:0/4168942884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c112330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009ca0 tx=0x7f2d1402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 -- 172.21.15.39:0/4168942884 shutdown_connections 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 --2- 172.21.15.39:0/4168942884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c112330 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 --2- 172.21.15.39:0/4168942884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 0x7f2d1c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 --2- 172.21.15.39:0/4168942884 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 -- 172.21.15.39:0/4168942884 >> 172.21.15.39:0/4168942884 conn(0x7f2d1c0ff3e0 msgr2=0x7f2d1c101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 -- 172.21.15.39:0/4168942884 shutdown_connections 2024-09-21T01:09:53.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.055+0000 7f2d20a00640 1 -- 172.21.15.39:0/4168942884 wait complete. 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.056+0000 7f2d20a00640 1 Processor -- start 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.056+0000 7f2d20a00640 1 -- start start 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d20a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d20a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 0x7f2d1c19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d20a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c1a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d20a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c115aa0 con 0x7f2d1c076100 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d20a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c115920 con 0x7f2d1c10fef0 2024-09-21T01:09:53.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d20a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c115c20 con 0x7f2d1c0757b0 2024-09-21T01:09:53.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:53.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c1a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:53.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42214/0 (socket says 172.21.15.39:42214) 2024-09-21T01:09:53.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 -- 172.21.15.39:0/118304338 learned_addr learned my addr 172.21.15.39:0/118304338 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:53.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1aa00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 0x7f2d1c19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 -- 172.21.15.39:0/118304338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 msgr2=0x7f2d1c1a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c1a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 -- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 msgr2=0x7f2d1c19c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 0x7f2d1c19c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1b400640 1 -- 172.21.15.39:0/118304338 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c1a23a0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1aa00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 0x7f2d1c19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.057+0000 7f2d1be00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.058+0000 7f2d1b400640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c19c430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00ea60 tx=0x7f2d0c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.058+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c00cde0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.058+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d0c0035f0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.058+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c1a2690 con 0x7f2d1c0757b0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.058+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c010690 con 0x7f2d1c0757b0 2024-09-21T01:09:53.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.058+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c1a8df0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.059+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d1c075bc0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.062+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2d0c0037a0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.063+0000 7f2d13e00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2d000b6700 0x7f2d000b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.063+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2d0c0e80c0 con 0x7f2d1c0757b0 2024-09-21T01:09:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.064+0000 7f2d1aa00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2d000b6700 0x7f2d000b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.064+0000 7f2d1aa00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2d000b6700 0x7f2d000b8c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c112210 tx=0x7f2d08005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.065+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2d0c0ae230 con 0x7f2d1c0757b0 2024-09-21T01:09:53.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 284043 5.56 MB/sec execute 140 sec latency 16.760 ms 2024-09-21T01:09:53.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.309+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d1c19d560 con 0x7f2d1c0757b0 2024-09-21T01:09:53.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.310+0000 7f2d13e00640 1 -- 172.21.15.39:0/118304338 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11911 (secure 0 0 0) 0x7f2d0c0b2610 con 0x7f2d1c0757b0 2024-09-21T01:09:53.311 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:09:53.311 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"btime":"2024-09-21T01:08:19:688675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:08:19.501727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:09:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2d000b6700 msgr2=0x7f2d000b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2d000b6700 0x7f2d000b8c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c112210 tx=0x7f2d08005f90 comp rx=0 tx=0).stop 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 msgr2=0x7f2d1c19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c19c430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00ea60 tx=0x7f2d0c00ef20 comp rx=0 tx=0).stop 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 shutdown_connections 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f2d000b6700 0x7f2d000b8c10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c10fef0 0x7f2d1c1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d1c076100 0x7f2d1c19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 --2- 172.21.15.39:0/118304338 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2d1c0757b0 0x7f2d1c19c430 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:53.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 >> 172.21.15.39:0/118304338 conn(0x7f2d1c0ff3e0 msgr2=0x7f2d1c109a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:53.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 shutdown_connections 2024-09-21T01:09:53.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:53.313+0000 7f2d20a00640 1 -- 172.21.15.39:0/118304338 wait complete. 2024-09-21T01:09:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:53 smithi039 ceph-mon[30415]: pgmap v580: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 586 KiB/s wr, 24 op/s 2024-09-21T01:09:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:53 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/118304338' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:53.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:53 smithi150 ceph-mon[34945]: pgmap v580: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 586 KiB/s wr, 24 op/s 2024-09-21T01:09:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:53 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/118304338' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:53 smithi160 ceph-mon[34696]: pgmap v580: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 586 KiB/s wr, 24 op/s 2024-09-21T01:09:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:53 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/118304338' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:09:54.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 285393 5.56 MB/sec execute 141 sec latency 22.064 ms 2024-09-21T01:09:54.314 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0' 2024-09-21T01:09:54.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0 2024-09-21T01:09:54.763 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:09:55.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 287558 5.61 MB/sec execute 142 sec latency 24.180 ms 2024-09-21T01:09:55.674 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:55 smithi039 ceph-mon[30415]: pgmap v581: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 610 KiB/s wr, 31 op/s 2024-09-21T01:09:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 -- 172.21.15.39:0/1867571188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158103990 msgr2=0x7f4158103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 --2- 172.21.15.39:0/1867571188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158103990 0x7f4158103da0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4148009900 tx=0x7f414802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 -- 172.21.15.39:0/1867571188 shutdown_connections 2024-09-21T01:09:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 --2- 172.21.15.39:0/1867571188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4158109c20 0x7f4158110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 --2- 172.21.15.39:0/1867571188 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41581092d0 0x7f41581096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 --2- 172.21.15.39:0/1867571188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158103990 0x7f4158103da0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 -- 172.21.15.39:0/1867571188 >> 172.21.15.39:0/1867571188 conn(0x7f41580ff3e0 msgr2=0x7f4158101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 -- 172.21.15.39:0/1867571188 shutdown_connections 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.672+0000 7f415d400640 1 -- 172.21.15.39:0/1867571188 wait complete. 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.673+0000 7f415d400640 1 Processor -- start 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 -- start start 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4158103990 0x7f415819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41581092d0 0x7f415819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 0x7f415819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4158113930 con 0x7f4158103990 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41581137b0 con 0x7f4158109c20 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415d400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4158113ab0 con 0x7f41581092d0 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415ca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 0x7f415819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f4157e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4158103990 0x7f415819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415ca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 0x7f415819fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35002/0 (socket says 172.21.15.39:35002) 2024-09-21T01:09:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f415ca00640 1 -- 172.21.15.39:0/2070356786 learned_addr learned my addr 172.21.15.39:0/2070356786 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.674+0000 7f4157400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41581092d0 0x7f415819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415ca00640 1 -- 172.21.15.39:0/2070356786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41581092d0 msgr2=0x7f415819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415ca00640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41581092d0 0x7f415819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415ca00640 1 -- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4158103990 msgr2=0x7f415819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415ca00640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4158103990 0x7f415819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415ca00640 1 -- 172.21.15.39:0/2070356786 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41581a0210 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415ca00640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 0x7f415819fae0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f415000eb10 tx=0x7f415000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f4154c00640 1 -- 172.21.15.39:0/2070356786 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150004b20 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41581a0500 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f4154c00640 1 -- 172.21.15.39:0/2070356786 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4150004cc0 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.675+0000 7f4154c00640 1 -- 172.21.15.39:0/2070356786 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150010670 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.676+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41581a6c60 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.680+0000 7f4165615740 1 Processor -- start 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.680+0000 7f4154c00640 1 -- 172.21.15.39:0/2070356786 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4150005020 con 0x7f4158109c20 2024-09-21T01:09:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 -- start start 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b9735b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b9736610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5578b96aba90 con 0x5578b9365990 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5578b969aec0 con 0x5578b936db70 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.681+0000 7f4165615740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5578b969ecd0 con 0x5578b938e850 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b9735b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b9735b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42232/0 (socket says 172.21.15.39:42232) 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 -- 172.21.15.39:0/1603794899 learned_addr learned my addr 172.21.15.39:0/1603794899 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b9736610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157e00640 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 -- 172.21.15.39:0/1603794899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 msgr2=0x5578b9736610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b9736610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 -- 172.21.15.39:0/1603794899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 msgr2=0x5578b97360d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97360d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 -- 172.21.15.39:0/1603794899 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578b96409d0 con 0x5578b938e850 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.682+0000 7f4157400640 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b9735b90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f41440097e0 tx=0x7f4144009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db63410034f77d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.683+0000 7f414d200640 1 -- 172.21.15.39:0/1603794899 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144004050 con 0x5578b938e850 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.683+0000 7f414d200640 1 -- 172.21.15.39:0/1603794899 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f414400fc70 con 0x5578b938e850 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.683+0000 7f414d200640 1 -- 172.21.15.39:0/1603794899 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41440038f0 con 0x5578b938e850 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.683+0000 7f4154c00640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f413c0b6670 0x7f413c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.683+0000 7f4154c00640 1 -- 172.21.15.39:0/2070356786 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f41500e80d0 con 0x7f4158109c20 2024-09-21T01:09:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.683+0000 7f4157e00640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f413c0b6670 0x7f413c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 -- 172.21.15.39:0/1603794899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 msgr2=0x5578b9735b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b9735b90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f41440097e0 tx=0x7f4144009770 comp rx=0 tx=0).stop 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 -- 172.21.15.39:0/1603794899 shutdown_connections 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b9736610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97360d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 --2- 172.21.15.39:0/1603794899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b9735b90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 -- 172.21.15.39:0/1603794899 >> 172.21.15.39:0/1603794899 conn(0x5578b9382c40 msgr2=0x5578b9733a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 -- 172.21.15.39:0/1603794899 shutdown_connections 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4157e00640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f413c0b6670 0x7f413c0b8b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4148005ab0 tx=0x7f4148005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.684+0000 7f4165615740 1 -- 172.21.15.39:0/1603794899 wait complete. 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.685+0000 7f4165615740 1 -- start start 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.685+0000 7f4165615740 1 Processor -- start 2024-09-21T01:09:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.685+0000 7f4165615740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9741680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.685+0000 7f4165615740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b936db70 0x5578b9741bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4165615740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b938e850 0x5578b974d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4165615740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5578b94fc7b0 con 0x5578b938e850 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4165615740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5578b963f650 con 0x5578b9365990 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4165615740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5578b9744370 con 0x5578b936db70 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4165615740 10 client.0 _ll_register_callbacks cb 0x5578b93a60a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9741680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9741680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35020/0 (socket says 172.21.15.39:35020) 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b936db70 0x5578b9741bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157e00640 1 -- 172.21.15.39:0/1322032094 learned_addr learned my addr 172.21.15.39:0/1322032094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f415ca00640 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b938e850 0x5578b974d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157400640 1 -- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 msgr2=0x5578b9741680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157400640 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9741680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157400640 1 -- 172.21.15.39:0/1322032094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b938e850 msgr2=0x5578b974d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157400640 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b938e850 0x5578b974d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.686+0000 7f4157400640 1 -- 172.21.15.39:0/1322032094 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578b9408ba0 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4157400640 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b936db70 0x5578b9741bc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4144004600 tx=0x7f4144009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=819e75540dda36c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4141600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4141600640 1 -- 172.21.15.39:0/1322032094 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41440293f0 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f414400fc70 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4141600640 1 -- 172.21.15.39:0/1322032094 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f414401f3f0 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4141600640 1 -- 172.21.15.39:0/1322032094 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f414402f690 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4165615740 10 client.25795 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.687+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5578b97502b0 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4141600640 1 -- 172.21.15.39:0/1322032094 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f414400fc70 con 0x5578b936db70 2024-09-21T01:09:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4141600640 1 -- 172.21.15.39:0/1322032094 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4144029590 con 0x5578b936db70 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4165615740 10 client.25795 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4165615740 10 client.25795 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4165615740 20 client.25795 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4165615740 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b934d1d0 0x5578b9753610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4165615740 4 client.25795 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578b96ad9e0 con 0x5578b934d1d0 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.688+0000 7f4157e00640 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b934d1d0 0x5578b9753610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.689+0000 7f4157e00640 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b934d1d0 0x5578b9753610 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62c867871b256fb server_cookie=b6b77984b43af80f in_seq=0 out_seq=0 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.689+0000 7f4141600640 10 client.25795 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4141600640 1 -- 172.21.15.39:0/1322032094 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5578b96ad9e0 con 0x5578b934d1d0 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4141600640 10 client.25795 handle_command_reply: tid=0 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 client.25795 shutdown 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b936db70 msgr2=0x5578b9741bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b936db70 0x5578b9741bc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4144004600 tx=0x7f4144009ef0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 shutdown_connections 2024-09-21T01:09:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b934d1d0 0x5578b9753610 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b938e850 0x5578b974d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b936db70 0x5578b9741bc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 --2- 172.21.15.39:0/1322032094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9741680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4141600640 0 client.25795 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 >> 172.21.15.39:0/1322032094 conn(0x5578b9382c40 msgr2=0x5578b9738d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 shutdown_connections 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.690+0000 7f4165615740 1 -- 172.21.15.39:0/1322032094 wait complete. 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.691+0000 7f4165615740 20 client.25795 trim_cache size 0 max 16384 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 Processor -- start 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 -- start start 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9737580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b936db70 0x5578b974d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b974daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f41581b3550 con 0x5578b936db70 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41581b0e20 con 0x5578b9365990 2024-09-21T01:09:55.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.713+0000 7f4165615740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f41581aec70 con 0x5578b938e850 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b936db70 0x5578b974d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f415ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b974daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f415ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b974daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42252/0 (socket says 172.21.15.39:42252) 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b936db70 0x5578b974d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51886/0 (socket says 172.21.15.39:51886) 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157400640 1 -- 172.21.15.39:0/2206294447 learned_addr learned my addr 172.21.15.39:0/2206294447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9737580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 -- 172.21.15.39:0/2206294447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 msgr2=0x5578b974daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b974daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 -- 172.21.15.39:0/2206294447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b936db70 msgr2=0x5578b974d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b936db70 0x5578b974d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 -- 172.21.15.39:0/2206294447 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578b9742740 con 0x5578b9365990 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.714+0000 7f4157e00640 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9737580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4148006010 tx=0x7f414803e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=622b4ab387bf1f54 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.715+0000 7f4143e00640 1 -- 172.21.15.39:0/2206294447 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148035040 con 0x5578b9365990 2024-09-21T01:09:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.715+0000 7f4143e00640 1 -- 172.21.15.39:0/2206294447 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41480035f0 con 0x5578b9365990 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.715+0000 7f4143e00640 1 -- 172.21.15.39:0/2206294447 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148032710 con 0x5578b9365990 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 -- 172.21.15.39:0/2206294447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 msgr2=0x5578b9737580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9737580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4148006010 tx=0x7f414803e000 comp rx=0 tx=0).stop 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 -- 172.21.15.39:0/2206294447 shutdown_connections 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b974daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b936db70 0x5578b974d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 --2- 172.21.15.39:0/2206294447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b9365990 0x5578b9737580 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 -- 172.21.15.39:0/2206294447 >> 172.21.15.39:0/2206294447 conn(0x5578b9382c40 msgr2=0x5578b9732060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 -- 172.21.15.39:0/2206294447 shutdown_connections 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.716+0000 7f4165615740 1 -- 172.21.15.39:0/2206294447 wait complete. 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.717+0000 7f4165615740 1 -- start start 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.717+0000 7f4165615740 1 Processor -- start 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b97b6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b97bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f41581af2e0 con 0x5578b9365990 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41581b0750 con 0x5578b936db70 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f41581af9b0 con 0x5578b938e850 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4165615740 10 client.0 _ll_register_callbacks cb 0x7f4144009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b97b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b97b6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35038/0 (socket says 172.21.15.39:35038) 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 -- 172.21.15.39:0/2562614594 learned_addr learned my addr 172.21.15.39:0/2562614594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f415ca00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b97bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 -- 172.21.15.39:0/2562614594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 msgr2=0x5578b97bb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b97bb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 -- 172.21.15.39:0/2562614594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 msgr2=0x5578b97b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97b5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.718+0000 7f4157400640 1 -- 172.21.15.39:0/2562614594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578b97bbb00 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4157400640 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b97b6590 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4144029990 tx=0x7f4144029db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a31effe783ec23ee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4141600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4141600640 1 -- 172.21.15.39:0/2562614594 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144036030 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41480035f0 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4141600640 1 -- 172.21.15.39:0/2562614594 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4144011240 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4141600640 1 -- 172.21.15.39:0/2562614594 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4144040050 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4165615740 10 client.25811 fetch_fsmap learned FSMap version 30 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5578b97c2eb0 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.719+0000 7f4141600640 1 -- 172.21.15.39:0/2562614594 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41480035f0 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4141600640 1 -- 172.21.15.39:0/2562614594 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f41440050b0 con 0x5578b936db70 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4165615740 10 client.25811 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:09:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4165615740 10 client.25811 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4165615740 20 client.25811 populate_metadata read hostname 'smithi039' 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4165615740 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b9322580 0x5578b97c5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4165615740 4 client.25811 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}] 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}) -- 0x5578b96a9490 con 0x5578b9322580 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.720+0000 7f4157e00640 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b9322580 0x5578b97c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.721+0000 7f4157e00640 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b9322580 0x5578b97c5760 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d341455f7783f3e server_cookie=722cdd51763a6b6b in_seq=0 out_seq=0 2024-09-21T01:09:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.721+0000 7f4141600640 10 client.25811 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:55 smithi150 ceph-mon[34945]: pgmap v581: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 610 KiB/s wr, 31 op/s 2024-09-21T01:09:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:55 smithi160 ceph-mon[34696]: pgmap v581: 129 pgs: 129 active+clean; 164 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 610 KiB/s wr, 31 op/s 2024-09-21T01:09:55.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4141600640 1 -- 172.21.15.39:0/2562614594 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5578b96a9490 con 0x5578b9322580 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4141600640 10 client.25811 handle_command_reply: tid=0 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 client.25811 shutdown 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 msgr2=0x5578b97b6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b97b6590 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4144029990 tx=0x7f4144029db0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 shutdown_connections 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5578b9322580 0x5578b97c5760 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5578b938e850 0x5578b97bb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578b936db70 0x5578b97b6590 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4141600640 0 client.25811 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 --2- 172.21.15.39:0/2562614594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578b9365990 0x5578b97b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 >> 172.21.15.39:0/2562614594 conn(0x5578b9382c40 msgr2=0x5578b9742910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 shutdown_connections 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.909+0000 7f4165615740 1 -- 172.21.15.39:0/2562614594 wait complete. 2024-09-21T01:09:55.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f4165615740 20 client.25811 trim_cache size 0 max 16384 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 30, 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 26, 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "4b9bf36c": { 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 26, 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:55.912 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.2, 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:55.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:09:55.914 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:09:55.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f413c0b6670 msgr2=0x7f413c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f413c0b6670 0x7f413c0b8b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4148005ab0 tx=0x7f4148005a20 comp rx=0 tx=0).stop 2024-09-21T01:09:55.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 msgr2=0x7f415819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 0x7f415819fae0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f415000eb10 tx=0x7f415000c2d0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 shutdown_connections 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f413c0b6670 0x7f413c0b8b80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4158109c20 0x7f415819fae0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41581092d0 0x7f415819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 --2- 172.21.15.39:0/2070356786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4158103990 0x7f415819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.910+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 >> 172.21.15.39:0/2070356786 conn(0x7f41580ff3e0 msgr2=0x7f4158101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.911+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 shutdown_connections 2024-09-21T01:09:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:09:55.911+0000 7f415d400640 1 -- 172.21.15.39:0/2070356786 wait complete. 2024-09-21T01:09:56.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.58 MB/sec execute 143 sec latency 434.254 ms 2024-09-21T01:09:56.938 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4b9bf36c', quiesce took 5.2 seconds. Will release after: 14.05 2024-09-21T01:09:57.157 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.54 MB/sec execute 144 sec latency 1434.363 ms 2024-09-21T01:09:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:57 smithi150 ceph-mon[34945]: pgmap v582: 129 pgs: 129 active+clean; 168 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 955 KiB/s wr, 74 op/s 2024-09-21T01:09:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:57 smithi160 ceph-mon[34696]: pgmap v582: 129 pgs: 129 active+clean; 168 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 955 KiB/s wr, 74 op/s 2024-09-21T01:09:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:57 smithi039 ceph-mon[30415]: pgmap v582: 129 pgs: 129 active+clean; 168 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 955 KiB/s wr, 74 op/s 2024-09-21T01:09:58.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.50 MB/sec execute 145 sec latency 2434.511 ms 2024-09-21T01:09:59.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.47 MB/sec execute 146 sec latency 3434.657 ms 2024-09-21T01:09:59.646 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:59 smithi039 ceph-mon[30415]: pgmap v583: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-09-21T01:09:59.646 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:09:59 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:09:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:59 smithi150 ceph-mon[34945]: pgmap v583: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-09-21T01:09:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:09:59 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:09:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:59 smithi160 ceph-mon[34696]: pgmap v583: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 101 op/s 2024-09-21T01:09:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:09:59 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:10:00.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.43 MB/sec execute 147 sec latency 4434.790 ms 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi150 to 176.0M 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi160 to 176.0M 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi150 to 184591769: error parsing value: Value '184591769' is below minimum 939524096 2024-09-21T01:10:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi160 to 184592486: error parsing value: Value '184592486' is below minimum 939524096 2024-09-21T01:10:00.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: overall HEALTH_OK 2024-09-21T01:10:00.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:10:00.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:10:00.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:10:00.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi150 to 176.0M 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi160 to 176.0M 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi150 to 184591769: error parsing value: Value '184591769' is below minimum 939524096 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi160 to 184592486: error parsing value: Value '184592486' is below minimum 939524096 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: overall HEALTH_OK 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:10:00.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:10:00.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi150 to 176.0M 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi160 to 176.0M 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi150 to 184591769: error parsing value: Value '184591769' is below minimum 939524096 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi160 to 184592486: error parsing value: Value '184592486' is below minimum 939524096 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: overall HEALTH_OK 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:10:01.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.39 MB/sec execute 148 sec latency 5434.901 ms 2024-09-21T01:10:01.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:01 smithi039 ceph-mon[30415]: pgmap v584: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 793 KiB/s wr, 95 op/s 2024-09-21T01:10:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:01 smithi150 ceph-mon[34945]: pgmap v584: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 793 KiB/s wr, 95 op/s 2024-09-21T01:10:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:01 smithi160 ceph-mon[34696]: pgmap v584: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 793 KiB/s wr, 95 op/s 2024-09-21T01:10:02.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.36 MB/sec execute 149 sec latency 6435.001 ms 2024-09-21T01:10:03.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.32 MB/sec execute 150 sec latency 7435.048 ms 2024-09-21T01:10:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:03 smithi150 ceph-mon[34945]: pgmap v585: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 877 KiB/s wr, 99 op/s 2024-09-21T01:10:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:03 smithi160 ceph-mon[34696]: pgmap v585: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 877 KiB/s wr, 99 op/s 2024-09-21T01:10:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:03 smithi039 ceph-mon[30415]: pgmap v585: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 877 KiB/s wr, 99 op/s 2024-09-21T01:10:04.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.29 MB/sec execute 151 sec latency 8435.110 ms 2024-09-21T01:10:05.158 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.25 MB/sec execute 152 sec latency 9435.176 ms 2024-09-21T01:10:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:05 smithi150 ceph-mon[34945]: pgmap v586: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 978 KiB/s wr, 118 op/s 2024-09-21T01:10:05.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:05 smithi160 ceph-mon[34696]: pgmap v586: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 978 KiB/s wr, 118 op/s 2024-09-21T01:10:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:05 smithi039 ceph-mon[30415]: pgmap v586: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 978 KiB/s wr, 118 op/s 2024-09-21T01:10:06.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.22 MB/sec execute 153 sec latency 10435.572 ms 2024-09-21T01:10:06.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:06 smithi150 ceph-mon[34945]: osdmap e91: 12 total, 12 up, 12 in 2024-09-21T01:10:06.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:06 smithi160 ceph-mon[34696]: osdmap e91: 12 total, 12 up, 12 in 2024-09-21T01:10:06.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:06 smithi039 ceph-mon[30415]: osdmap e91: 12 total, 12 up, 12 in 2024-09-21T01:10:07.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.18 MB/sec execute 154 sec latency 11435.769 ms 2024-09-21T01:10:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:07 smithi150 ceph-mon[34945]: pgmap v588: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 691 KiB/s wr, 80 op/s 2024-09-21T01:10:07.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:07 smithi160 ceph-mon[34696]: pgmap v588: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 691 KiB/s wr, 80 op/s 2024-09-21T01:10:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:07 smithi039 ceph-mon[30415]: pgmap v588: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 691 KiB/s wr, 80 op/s 2024-09-21T01:10:08.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.15 MB/sec execute 155 sec latency 12435.934 ms 2024-09-21T01:10:09.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.12 MB/sec execute 156 sec latency 13436.078 ms 2024-09-21T01:10:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:09 smithi150 ceph-mon[34945]: pgmap v589: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 486 KiB/s wr, 52 op/s 2024-09-21T01:10:09.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:09 smithi160 ceph-mon[34696]: pgmap v589: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 486 KiB/s wr, 52 op/s 2024-09-21T01:10:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:09 smithi039 ceph-mon[30415]: pgmap v589: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 486 KiB/s wr, 52 op/s 2024-09-21T01:10:10.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.08 MB/sec execute 157 sec latency 14436.221 ms 2024-09-21T01:10:10.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:10:11.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.05 MB/sec execute 158 sec latency 15436.296 ms 2024-09-21T01:10:11.383 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:10:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:11 smithi150 ceph-mon[34945]: pgmap v590: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 419 KiB/s wr, 51 op/s 2024-09-21T01:10:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:11 smithi160 ceph-mon[34696]: pgmap v590: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 419 KiB/s wr, 51 op/s 2024-09-21T01:10:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:11 smithi039 ceph-mon[30415]: pgmap v590: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 419 KiB/s wr, 51 op/s 2024-09-21T01:10:12.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 5.02 MB/sec execute 159 sec latency 16436.407 ms 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 -- 172.21.15.39:0/3951282135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc0757b0 msgr2=0x7f61bc075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 --2- 172.21.15.39:0/3951282135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc0757b0 0x7f61bc075bc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009900 tx=0x7f61b002a410 comp rx=0 tx=0).stop 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 -- 172.21.15.39:0/3951282135 shutdown_connections 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 --2- 172.21.15.39:0/3951282135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc1100e0 0x7f61bc112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 --2- 172.21.15.39:0/3951282135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61bc076100 0x7f61bc10fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 --2- 172.21.15.39:0/3951282135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc0757b0 0x7f61bc075bc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 -- 172.21.15.39:0/3951282135 >> 172.21.15.39:0/3951282135 conn(0x7f61bc0ff5d0 msgr2=0x7f61bc101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 -- 172.21.15.39:0/3951282135 shutdown_connections 2024-09-21T01:10:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.443+0000 7f61c3800640 1 -- 172.21.15.39:0/3951282135 wait complete. 2024-09-21T01:10:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 Processor -- start 2024-09-21T01:10:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 -- start start 2024-09-21T01:10:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 0x7f61bc19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 0x7f61bc19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61bc1100e0 0x7f61bc1a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc115ce0 con 0x7f61bc076100 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc115b60 con 0x7f61bc1100e0 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.445+0000 7f61c3800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc115e60 con 0x7f61bc0757b0 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 0x7f61bc19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 0x7f61bc19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 0x7f61bc19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36520/0 (socket says 172.21.15.39:36520) 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 -- 172.21.15.39:0/3602903092 learned_addr learned my addr 172.21.15.39:0/3602903092 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 0x7f61bc19cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34602/0 (socket says 172.21.15.39:34602) 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c1800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61bc1100e0 0x7f61bc1a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 -- 172.21.15.39:0/3602903092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61bc1100e0 msgr2=0x7f61bc1a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61bc1100e0 0x7f61bc1a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 -- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 msgr2=0x7f61bc19cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 0x7f61bc19cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 -- 172.21.15.39:0/3602903092 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc1a2660 con 0x7f61bc0757b0 2024-09-21T01:10:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61bbe00640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 0x7f61bc19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:10:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.446+0000 7f61c0e00640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 0x7f61bc19c6f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005ab0 tx=0x7f61b0003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.447+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b003f070 con 0x7f61bc0757b0 2024-09-21T01:10:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.447+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b0031b60 con 0x7f61bc0757b0 2024-09-21T01:10:12.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.447+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61bc1a28f0 con 0x7f61bc0757b0 2024-09-21T01:10:12.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.447+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0035030 con 0x7f61bc0757b0 2024-09-21T01:10:12.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.447+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61bc1a90b0 con 0x7f61bc0757b0 2024-09-21T01:10:12.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.448+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61bc075bc0 con 0x7f61bc0757b0 2024-09-21T01:10:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.451+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f61b0042070 con 0x7f61bc0757b0 2024-09-21T01:10:12.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.452+0000 7f61b9600640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61a40b6700 0x7f61a40b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:12.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.453+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7f61b0108a40 con 0x7f61bc0757b0 2024-09-21T01:10:12.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.453+0000 7f61bbe00640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61a40b6700 0x7f61a40b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:12.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.453+0000 7f61bbe00640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61a40b6700 0x7f61a40b8c10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f61bc1123d0 tx=0x7f61a8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:12.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.454+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f61b00ceb00 con 0x7f61bc0757b0 2024-09-21T01:10:12.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.696+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61bc19d820 con 0x7f61bc0757b0 2024-09-21T01:10:12.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.697+0000 7f61b9600640 1 -- 172.21.15.39:0/3602903092 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11911 (secure 0 0 0) 0x7f61b00d2ee0 con 0x7f61bc0757b0 2024-09-21T01:10:12.698 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:10:12.698 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"btime":"2024-09-21T01:08:19:688675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:08:19.501727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:10:12.698 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.699+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61a40b6700 msgr2=0x7f61a40b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.699+0000 7f61c3800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61a40b6700 0x7f61a40b8c10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f61bc1123d0 tx=0x7f61a8005f90 comp rx=0 tx=0).stop 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.699+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 msgr2=0x7f61bc19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.699+0000 7f61c3800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 0x7f61bc19c6f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005ab0 tx=0x7f61b0003c10 comp rx=0 tx=0).stop 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 shutdown_connections 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f61a40b6700 0x7f61a40b8c10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61bc1100e0 0x7f61bc1a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61bc076100 0x7f61bc19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 --2- 172.21.15.39:0/3602903092 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f61bc0757b0 0x7f61bc19c6f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 >> 172.21.15.39:0/3602903092 conn(0x7f61bc0ff5d0 msgr2=0x7f61bc109c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 shutdown_connections 2024-09-21T01:10:12.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:12.700+0000 7f61c3800640 1 -- 172.21.15.39:0/3602903092 wait complete. 2024-09-21T01:10:13.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 4.99 MB/sec execute 160 sec latency 17436.516 ms 2024-09-21T01:10:13.613 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 4b9bf36c --release --await' 2024-09-21T01:10:13.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 4b9bf36c --release --await 2024-09-21T01:10:13.807 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:13 smithi039 ceph-mon[30415]: pgmap v591: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 293 KiB/s wr, 42 op/s 2024-09-21T01:10:13.807 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:13 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3602903092' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:10:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:13 smithi150 ceph-mon[34945]: pgmap v591: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 293 KiB/s wr, 42 op/s 2024-09-21T01:10:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:13 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3602903092' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:10:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:13 smithi160 ceph-mon[34696]: pgmap v591: 129 pgs: 129 active+clean; 171 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 293 KiB/s wr, 42 op/s 2024-09-21T01:10:13.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:13 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3602903092' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:10:13.995 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:10:14.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288245 4.96 MB/sec execute 161 sec latency 18436.619 ms 2024-09-21T01:10:15.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 -- 172.21.15.39:0/902372467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc076100 msgr2=0x7f65bc10f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 --2- 172.21.15.39:0/902372467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc076100 0x7f65bc10f9d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009900 tx=0x7f65a802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 -- 172.21.15.39:0/902372467 shutdown_connections 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 --2- 172.21.15.39:0/902372467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc10ff10 0x7f65bc112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 --2- 172.21.15.39:0/902372467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc076100 0x7f65bc10f9d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 --2- 172.21.15.39:0/902372467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65bc0757b0 0x7f65bc075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 -- 172.21.15.39:0/902372467 >> 172.21.15.39:0/902372467 conn(0x7f65bc0ff440 msgr2=0x7f65bc101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 -- 172.21.15.39:0/902372467 shutdown_connections 2024-09-21T01:10:15.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.060+0000 7f65c0a00640 1 -- 172.21.15.39:0/902372467 wait complete. 2024-09-21T01:10:15.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.061+0000 7f65c0a00640 1 Processor -- start 2024-09-21T01:10:15.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.061+0000 7f65c0a00640 1 -- start start 2024-09-21T01:10:15.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65c0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 0x7f65bc19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65c0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65bc076100 0x7f65bc19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65c0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc10ff10 0x7f65bc1a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65c0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc115ac0 con 0x7f65bc0757b0 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65c0a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc115940 con 0x7f65bc10ff10 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65c0a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc115c40 con 0x7f65bc076100 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 0x7f65bc19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 0x7f65bc19c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58918/0 (socket says 172.21.15.39:58918) 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 -- 172.21.15.39:0/335547220 learned_addr learned my addr 172.21.15.39:0/335547220 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65baa00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65bc076100 0x7f65bc19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bbe00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc10ff10 0x7f65bc1a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 -- 172.21.15.39:0/335547220 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65bc076100 msgr2=0x7f65bc19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65bc076100 0x7f65bc19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 -- 172.21.15.39:0/335547220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc10ff10 msgr2=0x7f65bc1a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc10ff10 0x7f65bc1a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.062+0000 7f65bb400640 1 -- 172.21.15.39:0/335547220 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65bc1a2460 con 0x7f65bc0757b0 2024-09-21T01:10:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.063+0000 7f65bb400640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 0x7f65bc19c4f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00d950 tx=0x7f65ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.063+0000 7f65b3e00640 1 -- 172.21.15.39:0/335547220 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac017070 con 0x7f65bc0757b0 2024-09-21T01:10:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.063+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65bc1a2750 con 0x7f65bc0757b0 2024-09-21T01:10:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.063+0000 7f65b3e00640 1 -- 172.21.15.39:0/335547220 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65ac009c30 con 0x7f65bc0757b0 2024-09-21T01:10:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.063+0000 7f65b3e00640 1 -- 172.21.15.39:0/335547220 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00f620 con 0x7f65bc0757b0 2024-09-21T01:10:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.064+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65bc1a8eb0 con 0x7f65bc0757b0 2024-09-21T01:10:15.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 Processor -- start 2024-09-21T01:10:15.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65b3e00640 1 -- 172.21.15.39:0/335547220 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f65ac010460 con 0x7f65bc0757b0 2024-09-21T01:10:15.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 -- start start 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbc8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbc8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc800840 0x55aabcbc8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55aabcb3f290 con 0x55aabc821520 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55aabcb2e6c0 con 0x55aabc800840 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.067+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55aabcb324d0 con 0x55aabc7f8660 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbc8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbc8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58932/0 (socket says 172.21.15.39:58932) 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 -- 172.21.15.39:0/1557454773 learned_addr learned my addr 172.21.15.39:0/1557454773 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65baa00640 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc800840 0x55aabcbc8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bb400640 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbc8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 -- 172.21.15.39:0/1557454773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 msgr2=0x55aabcbc8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbc8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 -- 172.21.15.39:0/1557454773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc800840 msgr2=0x55aabcbc8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc800840 0x55aabcbc8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65bbe00640 1 -- 172.21.15.39:0/1557454773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aabcabb460 con 0x55aabc821520 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.068+0000 7f65baa00640 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc800840 0x55aabcbc8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbc8340 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009fd0 tx=0x7f65b40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11e4065cdb5e46b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65b3e00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65a00b6680 0x7f65a00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65a7e00640 1 -- 172.21.15.39:0/1557454773 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4004d20 con 0x55aabc821520 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65a7e00640 1 -- 172.21.15.39:0/1557454773 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65b4004050 con 0x55aabc821520 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65b3e00640 1 -- 172.21.15.39:0/335547220 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f65ac0e7720 con 0x7f65bc0757b0 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65a7e00640 1 -- 172.21.15.39:0/1557454773 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4005260 con 0x55aabc821520 2024-09-21T01:10:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.069+0000 7f65bb400640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65a00b6680 0x7f65a00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1557454773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 msgr2=0x55aabcbc8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65bb400640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65a00b6680 0x7f65a00b8b90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55aabcbc6fc0 tx=0x7f65ac0b4f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbc8340 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009fd0 tx=0x7f65b40118d0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1557454773 shutdown_connections 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc800840 0x55aabcbc8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbc8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1557454773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbc8340 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1557454773 >> 172.21.15.39:0/1557454773 conn(0x55aabc815910 msgr2=0x55aabcbc6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1557454773 shutdown_connections 2024-09-21T01:10:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.070+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1557454773 wait complete. 2024-09-21T01:10:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.071+0000 7f65c8bc4740 1 -- start start 2024-09-21T01:10:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.071+0000 7f65c8bc4740 1 Processor -- start 2024-09-21T01:10:15.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.071+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbd3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.071+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 0x55aabcbd43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.071+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbdfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55aabc98fbb0 con 0x55aabc800840 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55aabcac96c0 con 0x55aabc821520 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55aabcbd6b20 con 0x55aabc7f8660 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65c8bc4740 10 client.0 _ll_register_callbacks cb 0x55aabcae4490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 0x55aabcbd43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 0x55aabcbd43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58938/0 (socket says 172.21.15.39:58938) 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 -- 172.21.15.39:0/1589293907 learned_addr learned my addr 172.21.15.39:0/1589293907 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbd3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbdfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 -- 172.21.15.39:0/1589293907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 msgr2=0x55aabcbd3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbd3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 -- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 msgr2=0x55aabcbdfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbdfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65baa00640 1 -- 172.21.15.39:0/1589293907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aabc88cb00 con 0x55aabc800840 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.072+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbd3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65baa00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 0x55aabcbd43f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f65bc06b700 tx=0x7f65a8032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501bc000db475c0f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65a4c00640 1 -- 172.21.15.39:0/1589293907 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8035040 con 0x55aabc800840 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65b4004050 con 0x55aabc800840 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65a4c00640 1 -- 172.21.15.39:0/1589293907 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f65a80316a0 con 0x55aabc800840 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65a4c00640 1 -- 172.21.15.39:0/1589293907 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f65a804f050 con 0x55aabc800840 2024-09-21T01:10:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65c8bc4740 10 client.15798 fetch_fsmap learned FSMap version 30 2024-09-21T01:10:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.073+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55aabc903960 con 0x55aabc800840 2024-09-21T01:10:15.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65a4c00640 1 -- 172.21.15.39:0/1589293907 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65b4004050 con 0x55aabc800840 2024-09-21T01:10:15.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65a4c00640 1 -- 172.21.15.39:0/1589293907 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f65a8004750 con 0x55aabc800840 2024-09-21T01:10:15.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65c8bc4740 10 client.15798 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:10:15.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65c8bc4740 10 client.15798 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65c8bc4740 20 client.15798 populate_metadata read hostname 'smithi039' 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7dfea0 0x55aabcbe5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65c8bc4740 4 client.15798 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.074+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aabcb411e0 con 0x55aabc7dfea0 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.075+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7dfea0 0x55aabcbe5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.075+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7dfea0 0x55aabcbe5eb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f9769eedcffd1a server_cookie=76e79f16655e4237 in_seq=0 out_seq=0 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.075+0000 7f65a4c00640 10 client.15798 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.076+0000 7f65a4c00640 1 -- 172.21.15.39:0/1589293907 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55aabcb411e0 con 0x55aabc7dfea0 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.076+0000 7f65a4c00640 10 client.15798 handle_command_reply: tid=0 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.076+0000 7f65c8bc4740 1 client.15798 shutdown 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 msgr2=0x55aabcbd43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 0x55aabcbd43f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f65bc06b700 tx=0x7f65a8032e90 comp rx=0 tx=0).stop 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 shutdown_connections 2024-09-21T01:10:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7dfea0 0x55aabcbe5eb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbdfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc800840 0x55aabcbd43f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1589293907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc7f8660 0x55aabcbd3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 >> 172.21.15.39:0/1589293907 conn(0x55aabc815910 msgr2=0x55aabcbcb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65a4c00640 0 client.15798 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 shutdown_connections 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1589293907 wait complete. 2024-09-21T01:10:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.077+0000 7f65c8bc4740 20 client.15798 trim_cache size 0 max 16384 2024-09-21T01:10:15.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 Processor -- start 2024-09-21T01:10:15.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 -- start start 2024-09-21T01:10:15.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 0x55aabcbc62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbcd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbcdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1b1520 con 0x55aabc7f8660 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1aedf0 con 0x55aabc821520 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.095+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1acce0 con 0x55aabc800840 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 0x55aabcbc62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 0x55aabcbc62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58950/0 (socket says 172.21.15.39:58950) 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65bbe00640 1 -- 172.21.15.39:0/2583097523 learned_addr learned my addr 172.21.15.39:0/2583097523 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65baa00640 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbcd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65bbe00640 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbcdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65baa00640 1 -- 172.21.15.39:0/2583097523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 msgr2=0x55aabcbcdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65baa00640 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbcdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65baa00640 1 -- 172.21.15.39:0/2583097523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 msgr2=0x55aabcbc62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65baa00640 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 0x55aabcbc62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65baa00640 1 -- 172.21.15.39:0/2583097523 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aabcbc5960 con 0x55aabc800840 2024-09-21T01:10:15.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.096+0000 7f65bbe00640 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 0x55aabcbc62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.097+0000 7f65baa00640 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbcd5c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f65a8006010 tx=0x7f65a8005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70599adf5428de98 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.097+0000 7f65a7400640 1 -- 172.21.15.39:0/2583097523 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8058070 con 0x55aabc800840 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.097+0000 7f65a7400640 1 -- 172.21.15.39:0/2583097523 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a8031d10 con 0x55aabc800840 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.097+0000 7f65a7400640 1 -- 172.21.15.39:0/2583097523 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8035030 con 0x55aabc800840 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 -- 172.21.15.39:0/2583097523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 msgr2=0x55aabcbcd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbcd5c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f65a8006010 tx=0x7f65a8005ea0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 -- 172.21.15.39:0/2583097523 shutdown_connections 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc821520 0x55aabcbcdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbcd5c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/2583097523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc7f8660 0x55aabcbc62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 -- 172.21.15.39:0/2583097523 >> 172.21.15.39:0/2583097523 conn(0x55aabc815910 msgr2=0x55aabcbc5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:15.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 -- 172.21.15.39:0/2583097523 shutdown_connections 2024-09-21T01:10:15.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 -- 172.21.15.39:0/2583097523 wait complete. 2024-09-21T01:10:15.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 -- start start 2024-09-21T01:10:15.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.098+0000 7f65c8bc4740 1 Processor -- start 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc7f8660 0x55aabcbdc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbd6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbd72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1ad330 con 0x55aabc821520 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1ae720 con 0x55aabc7f8660 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1ad980 con 0x55aabc800840 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65c8bc4740 10 client.0 _ll_register_callbacks cb 0x7f65a8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbd72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65baa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbd6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbd72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58964/0 (socket says 172.21.15.39:58964) 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65bbe00640 1 -- 172.21.15.39:0/1800385864 learned_addr learned my addr 172.21.15.39:0/1800385864 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.099+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc7f8660 0x55aabcbdc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65baa00640 1 -- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc7f8660 msgr2=0x55aabcbdc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65baa00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc7f8660 0x55aabcbdc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65baa00640 1 -- 172.21.15.39:0/1800385864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 msgr2=0x55aabcbd72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65baa00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbd72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65baa00640 1 -- 172.21.15.39:0/1800385864 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aabcbd78e0 con 0x55aabc800840 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbd72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65baa00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbd6d60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f65a802a910 tx=0x7f65a8031750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c8db5da051f020 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65a4c00640 1 -- 172.21.15.39:0/1800385864 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8035070 con 0x55aabc800840 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.100+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a8031d10 con 0x55aabc800840 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.101+0000 7f65a4c00640 1 -- 172.21.15.39:0/1800385864 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f65a8004030 con 0x55aabc800840 2024-09-21T01:10:15.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.101+0000 7f65a4c00640 1 -- 172.21.15.39:0/1800385864 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f65a8067050 con 0x55aabc800840 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.101+0000 7f65c8bc4740 10 client.25825 fetch_fsmap learned FSMap version 30 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.101+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55aabcc51640 con 0x55aabc800840 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.101+0000 7f65a4c00640 1 -- 172.21.15.39:0/1800385864 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a8031d10 con 0x55aabc800840 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65a4c00640 1 -- 172.21.15.39:0/1800385864 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f65a805b020 con 0x55aabc800840 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65c8bc4740 10 client.25825 fetch_fsmap finished waiting for FSMap version 30 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65c8bc4740 10 client.25825 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65c8bc4740 20 client.25825 populate_metadata read hostname 'smithi039' 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7b5250 0x55aabcc53da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65c8bc4740 4 client.25825 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4b9bf36c", "release": true, "await": true}] 2024-09-21T01:10:15.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4b9bf36c", "release": true, "await": true}) -- 0x55aabcb3cc90 con 0x55aabc7b5250 2024-09-21T01:10:15.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.102+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7b5250 0x55aabcc53da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:10:15.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.103+0000 7f65bbe00640 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7b5250 0x55aabcc53da0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4aff3c9849fff66 server_cookie=1df1ff39fd8ed522 in_seq=0 out_seq=0 2024-09-21T01:10:15.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.103+0000 7f65a4c00640 10 client.25825 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65a4c00640 1 -- 172.21.15.39:0/1800385864 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55aabcb3cc90 con 0x55aabc7b5250 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65a4c00640 10 client.25825 handle_command_reply: tid=0 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 client.25825 shutdown 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 msgr2=0x55aabcbd6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbd6d60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f65a802a910 tx=0x7f65a8031750 comp rx=0 tx=0).stop 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 shutdown_connections 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55aabc7b5250 0x55aabcc53da0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aabc821520 0x55aabcbd72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55aabc800840 0x55aabcbd6d60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 --2- 172.21.15.39:0/1800385864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55aabc7f8660 0x55aabcbdc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65a4c00640 0 client.25825 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.108+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 >> 172.21.15.39:0/1800385864 conn(0x55aabc815910 msgr2=0x55aabcbc9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 shutdown_connections 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c8bc4740 1 -- 172.21.15.39:0/1800385864 wait complete. 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c8bc4740 20 client.25825 trim_cache size 0 max 16384 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 30, 2024-09-21T01:10:15.110 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 28, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "4b9bf36c": { 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 28, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.2, 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:10:15.111 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:10:15.112 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65a00b6680 msgr2=0x7f65a00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c0a00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65a00b6680 0x7f65a00b8b90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55aabcbc6fc0 tx=0x7f65ac0b4f90 comp rx=0 tx=0).stop 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 msgr2=0x7f65bc19c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.109+0000 7f65c0a00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 0x7f65bc19c4f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00d950 tx=0x7f65ac00de10 comp rx=0 tx=0).stop 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 shutdown_connections 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65a00b6680 0x7f65a00b8b90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65bc10ff10 0x7f65bc1a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65bc076100 0x7f65bc19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 --2- 172.21.15.39:0/335547220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65bc0757b0 0x7f65bc19c4f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:10:15.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 >> 172.21.15.39:0/335547220 conn(0x7f65bc0ff440 msgr2=0x7f65bc109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:10:15.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 shutdown_connections 2024-09-21T01:10:15.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:10:15.110+0000 7f65c0a00640 1 -- 172.21.15.39:0/335547220 wait complete. 2024-09-21T01:10:15.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288311 4.93 MB/sec execute 162 sec latency 19389.171 ms 2024-09-21T01:10:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:15 smithi039 ceph-mon[30415]: pgmap v592: 129 pgs: 129 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 184 KiB/s wr, 22 op/s 2024-09-21T01:10:15.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:15 smithi039 ceph-mon[30415]: osdmap e92: 12 total, 12 up, 12 in 2024-09-21T01:10:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:15 smithi150 ceph-mon[34945]: pgmap v592: 129 pgs: 129 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 184 KiB/s wr, 22 op/s 2024-09-21T01:10:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:15 smithi150 ceph-mon[34945]: osdmap e92: 12 total, 12 up, 12 in 2024-09-21T01:10:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:15 smithi160 ceph-mon[34696]: pgmap v592: 129 pgs: 129 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 184 KiB/s wr, 22 op/s 2024-09-21T01:10:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:15 smithi160 ceph-mon[34696]: osdmap e92: 12 total, 12 up, 12 in 2024-09-21T01:10:16.157 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4b9bf36c', total seconds elapsed: 24.4 2024-09-21T01:10:16.159 INFO:tasks.workunit.client.0.smithi039.stdout: 1 288767 4.92 MB/sec execute 163 sec latency 84.090 ms 2024-09-21T01:10:17.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 291016 4.97 MB/sec execute 164 sec latency 15.521 ms 2024-09-21T01:10:17.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:17 smithi039 ceph-mon[30415]: pgmap v594: 129 pgs: 129 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 182 KiB/s wr, 21 op/s 2024-09-21T01:10:17.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:17 smithi150 ceph-mon[34945]: pgmap v594: 129 pgs: 129 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 182 KiB/s wr, 21 op/s 2024-09-21T01:10:17.658 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:17 smithi160 ceph-mon[34696]: pgmap v594: 129 pgs: 129 active+clean; 170 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 182 KiB/s wr, 21 op/s 2024-09-21T01:10:18.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 292370 4.98 MB/sec execute 165 sec latency 17.965 ms 2024-09-21T01:10:19.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 293885 4.99 MB/sec execute 166 sec latency 44.976 ms 2024-09-21T01:10:19.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:19 smithi039 ceph-mon[30415]: pgmap v595: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 265 KiB/s wr, 44 op/s 2024-09-21T01:10:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:19 smithi150 ceph-mon[34945]: pgmap v595: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 265 KiB/s wr, 44 op/s 2024-09-21T01:10:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:19 smithi160 ceph-mon[34696]: pgmap v595: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 265 KiB/s wr, 44 op/s 2024-09-21T01:10:20.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 295492 5.02 MB/sec execute 167 sec latency 17.458 ms 2024-09-21T01:10:21.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 297453 5.05 MB/sec execute 168 sec latency 18.369 ms 2024-09-21T01:10:21.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:21 smithi039 ceph-mon[30415]: pgmap v596: 129 pgs: 129 active+clean; 173 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 452 KiB/s wr, 70 op/s 2024-09-21T01:10:21.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:21 smithi150 ceph-mon[34945]: pgmap v596: 129 pgs: 129 active+clean; 173 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 452 KiB/s wr, 70 op/s 2024-09-21T01:10:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:21 smithi160 ceph-mon[34696]: pgmap v596: 129 pgs: 129 active+clean; 173 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 452 KiB/s wr, 70 op/s 2024-09-21T01:10:22.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 299081 5.07 MB/sec execute 169 sec latency 21.712 ms 2024-09-21T01:10:23.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 301247 5.11 MB/sec execute 170 sec latency 30.342 ms 2024-09-21T01:10:23.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:23 smithi039 ceph-mon[30415]: pgmap v597: 129 pgs: 129 active+clean; 175 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 586 KiB/s wr, 89 op/s 2024-09-21T01:10:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:23 smithi150 ceph-mon[34945]: pgmap v597: 129 pgs: 129 active+clean; 175 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 586 KiB/s wr, 89 op/s 2024-09-21T01:10:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:23 smithi160 ceph-mon[34696]: pgmap v597: 129 pgs: 129 active+clean; 175 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 586 KiB/s wr, 89 op/s 2024-09-21T01:10:24.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 302813 5.12 MB/sec execute 171 sec latency 17.915 ms 2024-09-21T01:10:25.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 304290 5.14 MB/sec execute 172 sec latency 226.970 ms 2024-09-21T01:10:25.424 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:25 smithi039 ceph-mon[30415]: pgmap v598: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 838 KiB/s wr, 109 op/s 2024-09-21T01:10:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:25 smithi150 ceph-mon[34945]: pgmap v598: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 838 KiB/s wr, 109 op/s 2024-09-21T01:10:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:25 smithi160 ceph-mon[34696]: pgmap v598: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 838 KiB/s wr, 109 op/s 2024-09-21T01:10:26.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 304290 5.11 MB/sec execute 173 sec latency 1227.032 ms 2024-09-21T01:10:27.160 INFO:tasks.workunit.client.0.smithi039.stdout: 1 305523 5.13 MB/sec execute 174 sec latency 1364.188 ms 2024-09-21T01:10:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:27 smithi039 ceph-mon[30415]: pgmap v599: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-09-21T01:10:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:27 smithi150 ceph-mon[34945]: pgmap v599: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-09-21T01:10:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:27 smithi160 ceph-mon[34696]: pgmap v599: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-09-21T01:10:28.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 306757 5.13 MB/sec execute 175 sec latency 175.010 ms 2024-09-21T01:10:28.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:28 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:10:28.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:28 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:10:28.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:28 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:10:29.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 306757 5.10 MB/sec execute 176 sec latency 1175.157 ms 2024-09-21T01:10:29.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:29 smithi039 ceph-mon[30415]: pgmap v600: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 180 op/s 2024-09-21T01:10:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:29 smithi150 ceph-mon[34945]: pgmap v600: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 180 op/s 2024-09-21T01:10:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:29 smithi160 ceph-mon[34696]: pgmap v600: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 180 op/s 2024-09-21T01:10:30.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 308058 5.12 MB/sec execute 177 sec latency 1532.568 ms 2024-09-21T01:10:31.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 309593 5.13 MB/sec execute 178 sec latency 16.728 ms 2024-09-21T01:10:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:31 smithi039 ceph-mon[30415]: pgmap v601: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-09-21T01:10:31.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:31 smithi150 ceph-mon[34945]: pgmap v601: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-09-21T01:10:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:31 smithi160 ceph-mon[34696]: pgmap v601: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-09-21T01:10:32.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 310873 5.13 MB/sec execute 179 sec latency 419.010 ms 2024-09-21T01:10:33.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 312620 5.17 MB/sec execute 180 sec latency 13.355 ms 2024-09-21T01:10:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:33 smithi150 ceph-mon[34945]: pgmap v602: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 168 op/s 2024-09-21T01:10:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:33 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:10:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:33 smithi160 ceph-mon[34696]: pgmap v602: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 168 op/s 2024-09-21T01:10:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:33 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:10:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:33 smithi039 ceph-mon[30415]: pgmap v602: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 168 op/s 2024-09-21T01:10:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:33 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:10:34.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 313847 5.17 MB/sec execute 181 sec latency 173.075 ms 2024-09-21T01:10:35.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 315808 5.21 MB/sec execute 182 sec latency 385.791 ms 2024-09-21T01:10:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:35 smithi150 ceph-mon[34945]: pgmap v603: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-09-21T01:10:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:35 smithi160 ceph-mon[34696]: pgmap v603: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-09-21T01:10:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:35 smithi039 ceph-mon[30415]: pgmap v603: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-09-21T01:10:36.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 317157 5.22 MB/sec execute 183 sec latency 19.666 ms 2024-09-21T01:10:37.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 317396 5.19 MB/sec execute 184 sec latency 725.432 ms 2024-09-21T01:10:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:37 smithi150 ceph-mon[34945]: pgmap v604: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-09-21T01:10:37.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:37 smithi160 ceph-mon[34696]: pgmap v604: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-09-21T01:10:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:37 smithi039 ceph-mon[30415]: pgmap v604: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-09-21T01:10:38.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 317396 5.17 MB/sec execute 185 sec latency 1725.555 ms 2024-09-21T01:10:39.161 INFO:tasks.workunit.client.0.smithi039.stdout: 1 317396 5.14 MB/sec execute 186 sec latency 2725.697 ms 2024-09-21T01:10:39.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:39 smithi150 ceph-mon[34945]: pgmap v605: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-09-21T01:10:39.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:39 smithi160 ceph-mon[34696]: pgmap v605: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-09-21T01:10:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:39 smithi039 ceph-mon[30415]: pgmap v605: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-09-21T01:10:40.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 319355 5.18 MB/sec execute 187 sec latency 2940.163 ms 2024-09-21T01:10:41.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 320702 5.18 MB/sec execute 188 sec latency 43.264 ms 2024-09-21T01:10:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:41 smithi150 ceph-mon[34945]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-09-21T01:10:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:41 smithi160 ceph-mon[34696]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-09-21T01:10:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:41 smithi039 ceph-mon[30415]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-09-21T01:10:42.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 320938 5.16 MB/sec execute 189 sec latency 746.779 ms 2024-09-21T01:10:43.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 320938 5.13 MB/sec execute 190 sec latency 1746.911 ms 2024-09-21T01:10:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:43 smithi150 ceph-mon[34945]: pgmap v607: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-09-21T01:10:43.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:43 smithi160 ceph-mon[34696]: pgmap v607: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-09-21T01:10:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:43 smithi039 ceph-mon[30415]: pgmap v607: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-09-21T01:10:44.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 320938 5.10 MB/sec execute 191 sec latency 2747.059 ms 2024-09-21T01:10:45.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 322881 5.14 MB/sec execute 192 sec latency 2961.602 ms 2024-09-21T01:10:45.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:45 smithi039 ceph-mon[30415]: pgmap v608: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1019 KiB/s wr, 124 op/s 2024-09-21T01:10:45.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:45 smithi150 ceph-mon[34945]: pgmap v608: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1019 KiB/s wr, 124 op/s 2024-09-21T01:10:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:45 smithi160 ceph-mon[34696]: pgmap v608: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1019 KiB/s wr, 124 op/s 2024-09-21T01:10:46.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 324205 5.15 MB/sec execute 193 sec latency 17.261 ms 2024-09-21T01:10:47.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 324995 5.13 MB/sec execute 194 sec latency 548.125 ms 2024-09-21T01:10:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:47 smithi150 ceph-mon[34945]: pgmap v609: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-09-21T01:10:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:47 smithi160 ceph-mon[34696]: pgmap v609: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-09-21T01:10:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:47 smithi039 ceph-mon[30415]: pgmap v609: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-09-21T01:10:48.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 326889 5.17 MB/sec execute 195 sec latency 10.455 ms 2024-09-21T01:10:49.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 328029 5.17 MB/sec execute 196 sec latency 87.893 ms 2024-09-21T01:10:49.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:49 smithi150 ceph-mon[34945]: pgmap v610: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 922 KiB/s wr, 107 op/s 2024-09-21T01:10:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:49 smithi160 ceph-mon[34696]: pgmap v610: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 922 KiB/s wr, 107 op/s 2024-09-21T01:10:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:49 smithi039 ceph-mon[30415]: pgmap v610: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 922 KiB/s wr, 107 op/s 2024-09-21T01:10:50.162 INFO:tasks.workunit.client.0.smithi039.stdout: 1 329621 5.19 MB/sec execute 197 sec latency 442.223 ms 2024-09-21T01:10:51.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 330111 5.18 MB/sec execute 198 sec latency 722.572 ms 2024-09-21T01:10:51.566 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:51 smithi039 ceph-mon[30415]: pgmap v611: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 764 KiB/s wr, 84 op/s 2024-09-21T01:10:51.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:51 smithi150 ceph-mon[34945]: pgmap v611: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 764 KiB/s wr, 84 op/s 2024-09-21T01:10:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:51 smithi160 ceph-mon[34696]: pgmap v611: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 764 KiB/s wr, 84 op/s 2024-09-21T01:10:52.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 330552 5.16 MB/sec execute 199 sec latency 1315.450 ms 2024-09-21T01:10:53.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 331577 5.16 MB/sec execute 200 sec latency 307.835 ms 2024-09-21T01:10:53.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:53 smithi150 ceph-mon[34945]: pgmap v612: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-09-21T01:10:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:53 smithi160 ceph-mon[34696]: pgmap v612: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-09-21T01:10:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:53 smithi039 ceph-mon[30415]: pgmap v612: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-09-21T01:10:54.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 331577 5.13 MB/sec execute 201 sec latency 1307.922 ms 2024-09-21T01:10:55.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 333547 5.17 MB/sec execute 202 sec latency 1522.875 ms 2024-09-21T01:10:55.747 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:55 smithi039 ceph-mon[30415]: pgmap v613: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 115 op/s 2024-09-21T01:10:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:55 smithi150 ceph-mon[34945]: pgmap v613: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 115 op/s 2024-09-21T01:10:55.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:55 smithi160 ceph-mon[34696]: pgmap v613: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 115 op/s 2024-09-21T01:10:56.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 334022 5.15 MB/sec execute 203 sec latency 591.706 ms 2024-09-21T01:10:57.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 334150 5.13 MB/sec execute 204 sec latency 1464.584 ms 2024-09-21T01:10:57.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:57 smithi150 ceph-mon[34945]: pgmap v614: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-09-21T01:10:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:57 smithi160 ceph-mon[34696]: pgmap v614: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-09-21T01:10:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:57 smithi039 ceph-mon[30415]: pgmap v614: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-09-21T01:10:58.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 335120 5.13 MB/sec execute 205 sec latency 383.607 ms 2024-09-21T01:10:59.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 335120 5.10 MB/sec execute 206 sec latency 1383.755 ms 2024-09-21T01:10:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:10:59 smithi150 ceph-mon[34945]: pgmap v615: 129 pgs: 129 active+clean; 212 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-09-21T01:10:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:10:59 smithi160 ceph-mon[34696]: pgmap v615: 129 pgs: 129 active+clean; 212 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-09-21T01:10:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:10:59 smithi039 ceph-mon[30415]: pgmap v615: 129 pgs: 129 active+clean; 212 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-09-21T01:11:00.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 336754 5.12 MB/sec execute 207 sec latency 1739.975 ms 2024-09-21T01:11:00.742 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:00 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:11:00.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:00 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:11:00.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:00 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:11:01.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 337524 5.12 MB/sec execute 208 sec latency 239.656 ms 2024-09-21T01:11:01.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:01 smithi039 ceph-mon[30415]: pgmap v616: 129 pgs: 129 active+clean; 212 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-09-21T01:11:01.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:11:01.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:11:01.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:11:01.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:01 smithi150 ceph-mon[34945]: pgmap v616: 129 pgs: 129 active+clean; 212 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-09-21T01:11:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:11:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:11:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:11:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:01 smithi160 ceph-mon[34696]: pgmap v616: 129 pgs: 129 active+clean; 212 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-09-21T01:11:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:01 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:11:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:01 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:11:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:01 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:11:02.163 INFO:tasks.workunit.client.0.smithi039.stdout: 1 338379 5.12 MB/sec execute 209 sec latency 627.430 ms 2024-09-21T01:11:03.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 338670 5.10 MB/sec execute 210 sec latency 635.477 ms 2024-09-21T01:11:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:03 smithi150 ceph-mon[34945]: pgmap v617: 129 pgs: 129 active+clean; 214 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 164 op/s 2024-09-21T01:11:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:03 smithi160 ceph-mon[34696]: pgmap v617: 129 pgs: 129 active+clean; 214 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 164 op/s 2024-09-21T01:11:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:03 smithi039 ceph-mon[30415]: pgmap v617: 129 pgs: 129 active+clean; 214 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 164 op/s 2024-09-21T01:11:04.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 338670 5.07 MB/sec execute 211 sec latency 1635.615 ms 2024-09-21T01:11:05.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 340591 5.11 MB/sec execute 212 sec latency 1848.694 ms 2024-09-21T01:11:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:05 smithi150 ceph-mon[34945]: pgmap v618: 129 pgs: 129 active+clean; 211 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:11:05.823 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:05 smithi039 ceph-mon[30415]: pgmap v618: 129 pgs: 129 active+clean; 211 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:11:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:05 smithi160 ceph-mon[34696]: pgmap v618: 129 pgs: 129 active+clean; 211 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:11:06.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 341270 5.09 MB/sec execute 213 sec latency 185.213 ms 2024-09-21T01:11:06.651 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:06 smithi039 ceph-mon[30415]: osdmap e93: 12 total, 12 up, 12 in 2024-09-21T01:11:06.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:06 smithi150 ceph-mon[34945]: osdmap e93: 12 total, 12 up, 12 in 2024-09-21T01:11:06.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:06 smithi160 ceph-mon[34696]: osdmap e93: 12 total, 12 up, 12 in 2024-09-21T01:11:07.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 341432 5.07 MB/sec execute 214 sec latency 744.389 ms 2024-09-21T01:11:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:07 smithi150 ceph-mon[34945]: pgmap v620: 129 pgs: 129 active+clean; 214 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 174 op/s 2024-09-21T01:11:07.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:07 smithi160 ceph-mon[34696]: pgmap v620: 129 pgs: 129 active+clean; 214 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 174 op/s 2024-09-21T01:11:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:07 smithi039 ceph-mon[30415]: pgmap v620: 129 pgs: 129 active+clean; 214 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 174 op/s 2024-09-21T01:11:08.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 342210 5.07 MB/sec execute 215 sec latency 591.201 ms 2024-09-21T01:11:09.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 342210 5.04 MB/sec execute 216 sec latency 1591.356 ms 2024-09-21T01:11:09.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:09 smithi150 ceph-mon[34945]: pgmap v621: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:11:09.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:09 smithi160 ceph-mon[34696]: pgmap v621: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:11:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:09 smithi039 ceph-mon[30415]: pgmap v621: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:11:10.164 INFO:tasks.workunit.client.0.smithi039.stdout: 1 344172 5.08 MB/sec execute 217 sec latency 1800.657 ms 2024-09-21T01:11:11.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 345057 5.07 MB/sec execute 218 sec latency 246.274 ms 2024-09-21T01:11:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:11 smithi150 ceph-mon[34945]: pgmap v622: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 147 op/s 2024-09-21T01:11:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:11 smithi160 ceph-mon[34696]: pgmap v622: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 147 op/s 2024-09-21T01:11:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:11 smithi039 ceph-mon[30415]: pgmap v622: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 147 op/s 2024-09-21T01:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 345759 5.06 MB/sec execute 219 sec latency 534.792 ms 2024-09-21T01:11:13.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 345759 5.04 MB/sec execute 220 sec latency 1250.499 ms 2024-09-21T01:11:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:13 smithi150 ceph-mon[34945]: pgmap v623: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 135 op/s 2024-09-21T01:11:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:13 smithi160 ceph-mon[34696]: pgmap v623: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 135 op/s 2024-09-21T01:11:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:13 smithi039 ceph-mon[30415]: pgmap v623: 129 pgs: 129 active+clean; 217 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 135 op/s 2024-09-21T01:11:14.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 345759 5.02 MB/sec execute 221 sec latency 2250.653 ms 2024-09-21T01:11:15.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 347693 5.05 MB/sec execute 222 sec latency 2493.388 ms 2024-09-21T01:11:15.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:15 smithi039 ceph-mon[30415]: pgmap v624: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 141 op/s 2024-09-21T01:11:15.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:15 smithi039 ceph-mon[30415]: osdmap e94: 12 total, 12 up, 12 in 2024-09-21T01:11:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:15 smithi150 ceph-mon[34945]: pgmap v624: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 141 op/s 2024-09-21T01:11:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:15 smithi150 ceph-mon[34945]: osdmap e94: 12 total, 12 up, 12 in 2024-09-21T01:11:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:15 smithi160 ceph-mon[34696]: pgmap v624: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 141 op/s 2024-09-21T01:11:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:15 smithi160 ceph-mon[34696]: osdmap e94: 12 total, 12 up, 12 in 2024-09-21T01:11:16.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 348868 5.04 MB/sec execute 223 sec latency 118.589 ms 2024-09-21T01:11:17.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 349302 5.03 MB/sec execute 224 sec latency 444.589 ms 2024-09-21T01:11:17.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:17 smithi039 ceph-mon[30415]: pgmap v626: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-09-21T01:11:17.658 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:17 smithi160 ceph-mon[34696]: pgmap v626: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-09-21T01:11:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:17 smithi150 ceph-mon[34945]: pgmap v626: 129 pgs: 129 active+clean; 218 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-09-21T01:11:18.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 349302 5.01 MB/sec execute 225 sec latency 1162.265 ms 2024-09-21T01:11:19.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 349302 4.99 MB/sec execute 226 sec latency 2162.414 ms 2024-09-21T01:11:19.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:19 smithi039 ceph-mon[30415]: pgmap v627: 129 pgs: 129 active+clean; 224 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-09-21T01:11:19.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:19 smithi150 ceph-mon[34945]: pgmap v627: 129 pgs: 129 active+clean; 224 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-09-21T01:11:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:19 smithi160 ceph-mon[34696]: pgmap v627: 129 pgs: 129 active+clean; 224 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-09-21T01:11:20.165 INFO:tasks.workunit.client.0.smithi039.stdout: 1 350971 5.01 MB/sec execute 227 sec latency 2516.894 ms 2024-09-21T01:11:21.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 352378 5.02 MB/sec execute 228 sec latency 21.475 ms 2024-09-21T01:11:21.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:21 smithi039 ceph-mon[30415]: pgmap v628: 129 pgs: 129 active+clean; 224 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 131 op/s 2024-09-21T01:11:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:21 smithi150 ceph-mon[34945]: pgmap v628: 129 pgs: 129 active+clean; 224 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 131 op/s 2024-09-21T01:11:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:21 smithi160 ceph-mon[34696]: pgmap v628: 129 pgs: 129 active+clean; 224 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 131 op/s 2024-09-21T01:11:22.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 352379 4.99 MB/sec execute 229 sec latency 954.944 ms 2024-09-21T01:11:23.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 352850 4.99 MB/sec execute 230 sec latency 1064.244 ms 2024-09-21T01:11:23.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:23 smithi039 ceph-mon[30415]: pgmap v629: 129 pgs: 129 active+clean; 229 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 136 op/s 2024-09-21T01:11:23.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:23 smithi150 ceph-mon[34945]: pgmap v629: 129 pgs: 129 active+clean; 229 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 136 op/s 2024-09-21T01:11:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:23 smithi160 ceph-mon[34696]: pgmap v629: 129 pgs: 129 active+clean; 229 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 136 op/s 2024-09-21T01:11:24.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 352850 4.96 MB/sec execute 231 sec latency 1590.548 ms 2024-09-21T01:11:25.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 354764 5.00 MB/sec execute 232 sec latency 1804.699 ms 2024-09-21T01:11:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:25 smithi150 ceph-mon[34945]: pgmap v630: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-09-21T01:11:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:25 smithi160 ceph-mon[34696]: pgmap v630: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-09-21T01:11:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:25 smithi039 ceph-mon[30415]: pgmap v630: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-09-21T01:11:26.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 356153 5.00 MB/sec execute 233 sec latency 17.965 ms 2024-09-21T01:11:27.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 356392 4.98 MB/sec execute 234 sec latency 424.508 ms 2024-09-21T01:11:27.658 INFO:tasks.quiescer.fs.[cephfs]:Total duration (42.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:11:27.658 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.05 2024-09-21T01:11:27.659 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:11:27.717 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:27 smithi039 ceph-mon[30415]: pgmap v631: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 101 op/s 2024-09-21T01:11:27.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:27 smithi150 ceph-mon[34945]: pgmap v631: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 101 op/s 2024-09-21T01:11:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:27 smithi160 ceph-mon[34696]: pgmap v631: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 101 op/s 2024-09-21T01:11:28.055 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:11:28.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 356392 4.96 MB/sec execute 235 sec latency 1082.604 ms 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 -- 172.21.15.39:0/3128179361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 msgr2=0x7f022c105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 --2- 172.21.15.39:0/3128179361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c105fc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0220009900 tx=0x7f022002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 -- 172.21.15.39:0/3128179361 shutdown_connections 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 --2- 172.21.15.39:0/3128179361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 0x7f022c112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 --2- 172.21.15.39:0/3128179361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 --2- 172.21.15.39:0/3128179361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c105fc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 -- 172.21.15.39:0/3128179361 >> 172.21.15.39:0/3128179361 conn(0x7f022c101680 msgr2=0x7f022c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 -- 172.21.15.39:0/3128179361 shutdown_connections 2024-09-21T01:11:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.749+0000 7f0233000640 1 -- 172.21.15.39:0/3128179361 wait complete. 2024-09-21T01:11:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.750+0000 7f0233000640 1 Processor -- start 2024-09-21T01:11:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.750+0000 7f0233000640 1 -- start start 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0233000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0233000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c19cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0233000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 0x7f022c1a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0233000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f022c115b00 con 0x7f022c10b4f0 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0233000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f022c115980 con 0x7f022c10be40 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0233000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f022c115c80 con 0x7f022c105bb0 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0231200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c19cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0231c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0231200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c19cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37788/0 (socket says 172.21.15.39:37788) 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0231200640 1 -- 172.21.15.39:0/4096143782 learned_addr learned my addr 172.21.15.39:0/4096143782 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.751+0000 7f0232600640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 0x7f022c1a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0232600640 1 -- 172.21.15.39:0/4096143782 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 msgr2=0x7f022c19c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0232600640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c19c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0232600640 1 -- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 msgr2=0x7f022c19cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0232600640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c19cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0232600640 1 -- 172.21.15.39:0/4096143782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022c1a24a0 con 0x7f022c10be40 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0231200640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c19cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0231c00640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c19c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0232600640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 0x7f022c1a1d70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0224009fd0 tx=0x7f022400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022400ce20 con 0x7f022c10be40 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0224005020 con 0x7f022c10be40 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022c1a2790 con 0x7f022c10be40 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0224010690 con 0x7f022c10be40 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.752+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f022c1a8ef0 con 0x7f022c10be40 2024-09-21T01:11:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.754+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f022c105fc0 con 0x7f022c10be40 2024-09-21T01:11:28.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.756+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f02240054d0 con 0x7f022c10be40 2024-09-21T01:11:28.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.758+0000 7f022a000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f02140b6680 0x7f02140b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:28.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.758+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f02240e8200 con 0x7f022c10be40 2024-09-21T01:11:28.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.758+0000 7f0231c00640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f02140b6680 0x7f02140b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:28.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.759+0000 7f0231c00640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f02140b6680 0x7f02140b8b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0220031440 tx=0x7f0220032ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:28.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.760+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f02240ae310 con 0x7f022c10be40 2024-09-21T01:11:28.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.997+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f022c19d660 con 0x7f022c10be40 2024-09-21T01:11:28.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:28.998+0000 7f022a000640 1 -- 172.21.15.39:0/4096143782 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11914 (secure 0 0 0) 0x7f022c19d660 con 0x7f022c10be40 2024-09-21T01:11:29.000 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:11:29.000 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":32,"btime":"2024-09-21T01:10:32:401759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:10:31.978927+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:11:29.000 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 32 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f02140b6680 msgr2=0x7f02140b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f02140b6680 0x7f02140b8b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0220031440 tx=0x7f0220032ed0 comp rx=0 tx=0).stop 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 msgr2=0x7f022c1a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 0x7f022c1a1d70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0224009fd0 tx=0x7f022400ef10 comp rx=0 tx=0).stop 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 shutdown_connections 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f02140b6680 0x7f02140b8b90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c10be40 0x7f022c1a1d70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f022c10b4f0 0x7f022c19cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 --2- 172.21.15.39:0/4096143782 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f022c105bb0 0x7f022c19c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.001+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 >> 172.21.15.39:0/4096143782 conn(0x7f022c101680 msgr2=0x7f022c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:29.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.002+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 shutdown_connections 2024-09-21T01:11:29.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:29.002+0000 7f0233000640 1 -- 172.21.15.39:0/4096143782 wait complete. 2024-09-21T01:11:29.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 356392 4.94 MB/sec execute 236 sec latency 2082.728 ms 2024-09-21T01:11:29.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:29 smithi150 ceph-mon[34945]: pgmap v632: 129 pgs: 129 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 132 op/s 2024-09-21T01:11:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:29 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4096143782' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:29 smithi160 ceph-mon[34696]: pgmap v632: 129 pgs: 129 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 132 op/s 2024-09-21T01:11:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:29 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4096143782' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:29 smithi039 ceph-mon[30415]: pgmap v632: 129 pgs: 129 active+clean; 228 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 132 op/s 2024-09-21T01:11:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:29 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4096143782' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:30.109 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0' 2024-09-21T01:11:30.110 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0 2024-09-21T01:11:30.166 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358049 4.96 MB/sec execute 237 sec latency 2449.454 ms 2024-09-21T01:11:30.516 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:11:31.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.96 MB/sec execute 238 sec latency 401.195 ms 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 -- 172.21.15.39:0/1138441499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730075730 msgr2=0x7fc730075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 --2- 172.21.15.39:0/1138441499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730075730 0x7fc730075b40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc724009900 tx=0x7fc72402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 -- 172.21.15.39:0/1138441499 shutdown_connections 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 --2- 172.21.15.39:0/1138441499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc730110150 0x7fc730112570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 --2- 172.21.15.39:0/1138441499 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730076080 0x7fc73010fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 --2- 172.21.15.39:0/1138441499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730075730 0x7fc730075b40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 -- 172.21.15.39:0/1138441499 >> 172.21.15.39:0/1138441499 conn(0x7fc7300ff5a0 msgr2=0x7fc7301019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 -- 172.21.15.39:0/1138441499 shutdown_connections 2024-09-21T01:11:31.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.336+0000 7fc737e00640 1 -- 172.21.15.39:0/1138441499 wait complete. 2024-09-21T01:11:31.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.337+0000 7fc737e00640 1 Processor -- start 2024-09-21T01:11:31.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 -- start start 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc730075730 0x7fc73019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 0x7fc73019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730110150 0x7fc7301a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc730115ce0 con 0x7fc730076080 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc730115b60 con 0x7fc730075730 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc737e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc730115e60 con 0x7fc730110150 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.338+0000 7fc734a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 0x7fc73019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 0x7fc73019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37798/0 (socket says 172.21.15.39:37798) 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 -- 172.21.15.39:0/2450023193 learned_addr learned my addr 172.21.15.39:0/2450023193 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:31.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc735e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730110150 0x7fc7301a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc735400640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc730075730 0x7fc73019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 -- 172.21.15.39:0/2450023193 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730110150 msgr2=0x7fc7301a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730110150 0x7fc7301a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 -- 172.21.15.39:0/2450023193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc730075730 msgr2=0x7fc73019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc730075730 0x7fc73019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 -- 172.21.15.39:0/2450023193 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7301a2660 con 0x7fc730076080 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc735e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730110150 0x7fc7301a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.339+0000 7fc734a00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 0x7fc73019cc90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00e970 tx=0x7fc71c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.340+0000 7fc72e000640 1 -- 172.21.15.39:0/2450023193 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c00cd80 con 0x7fc730076080 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.340+0000 7fc72e000640 1 -- 172.21.15.39:0/2450023193 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71c0035f0 con 0x7fc730076080 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.340+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7301a2950 con 0x7fc730076080 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.340+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7301a90b0 con 0x7fc730076080 2024-09-21T01:11:31.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.340+0000 7fc72e000640 1 -- 172.21.15.39:0/2450023193 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c010690 con 0x7fc730076080 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.343+0000 7fc73ea8b740 1 Processor -- start 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 -- start start 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc72e000640 1 -- 172.21.15.39:0/2450023193 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fc71c010830 con 0x7fc730076080 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3ef2d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef2def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef2e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562e3eea3a90 con 0x562e3eb65b70 2024-09-21T01:11:31.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562e3ee92ec0 con 0x562e3eb5d990 2024-09-21T01:11:31.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x562e3ee96cd0 con 0x562e3eb86850 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc735e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3ef2d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc735e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3ef2d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39280/0 (socket says 172.21.15.39:39280) 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.344+0000 7fc735e00640 1 -- 172.21.15.39:0/3737735067 learned_addr learned my addr 172.21.15.39:0/3737735067 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735400640 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef2def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc734a00640 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef2e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735e00640 1 -- 172.21.15.39:0/3737735067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 msgr2=0x562e3ef2def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735e00640 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef2def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735e00640 1 -- 172.21.15.39:0/3737735067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 msgr2=0x562e3ef2e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735e00640 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef2e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735e00640 1 -- 172.21.15.39:0/3737735067 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3ee389d0 con 0x562e3eb86850 2024-09-21T01:11:31.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc734a00640 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef2e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:11:31.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.345+0000 7fc735e00640 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3ef2d9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc728011790 tx=0x7fc728011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74c6beed755ed6b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:31.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.346+0000 7fc722600640 1 -- 172.21.15.39:0/3737735067 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728003c30 con 0x562e3eb86850 2024-09-21T01:11:31.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.346+0000 7fc722600640 1 -- 172.21.15.39:0/3737735067 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72800fe50 con 0x562e3eb86850 2024-09-21T01:11:31.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.346+0000 7fc722600640 1 -- 172.21.15.39:0/3737735067 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7280193e0 con 0x562e3eb86850 2024-09-21T01:11:31.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.346+0000 7fc72e000640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7180b6680 0x7fc7180b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.346+0000 7fc73ea8b740 1 -- 172.21.15.39:0/3737735067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 msgr2=0x562e3ef2d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.346+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3ef2d9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc728011790 tx=0x7fc728011c50 comp rx=0 tx=0).stop 2024-09-21T01:11:31.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 -- 172.21.15.39:0/3737735067 shutdown_connections 2024-09-21T01:11:31.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef2e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef2def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/3737735067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3ef2d9b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 -- 172.21.15.39:0/3737735067 >> 172.21.15.39:0/3737735067 conn(0x562e3eb7ac40 msgr2=0x562e3ef2b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc72e000640 1 -- 172.21.15.39:0/2450023193 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc71c0e7c90 con 0x7fc730076080 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc735400640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7180b6680 0x7fc7180b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 -- 172.21.15.39:0/3737735067 shutdown_connections 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.347+0000 7fc73ea8b740 1 -- 172.21.15.39:0/3737735067 wait complete. 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.348+0000 7fc73ea8b740 1 -- start start 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.348+0000 7fc735400640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7180b6680 0x7fc7180b8b90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc724005ab0 tx=0x7fc724005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:31.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.348+0000 7fc73ea8b740 1 Processor -- start 2024-09-21T01:11:31.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.348+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef39200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb65b70 0x562e3ef39780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb86850 0x562e3ef454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562e3ecf47b0 con 0x562e3eb86850 2024-09-21T01:11:31.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562e3ee37650 con 0x562e3eb5d990 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x562e3ef3c190 con 0x562e3eb65b70 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc73ea8b740 10 client.0 _ll_register_callbacks cb 0x562e3eb9e0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef39200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc734a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb65b70 0x562e3ef39780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef39200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:55104/0 (socket says 172.21.15.39:55104) 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 -- 172.21.15.39:0/1101541516 learned_addr learned my addr 172.21.15.39:0/1101541516 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735e00640 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb86850 0x562e3ef454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 -- 172.21.15.39:0/1101541516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb65b70 msgr2=0x562e3ef39780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb65b70 0x562e3ef39780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 -- 172.21.15.39:0/1101541516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb86850 msgr2=0x562e3ef454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb86850 0x562e3ef454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.349+0000 7fc735400640 1 -- 172.21.15.39:0/1101541516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3ec00ba0 con 0x562e3eb5d990 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.350+0000 7fc735400640 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef39200 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc724009900 tx=0x7fc724009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d927eddab445418b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.350+0000 7fc716a00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.350+0000 7fc716a00640 1 -- 172.21.15.39:0/1101541516 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72404e070 con 0x562e3eb5d990 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.350+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc72800fe50 con 0x562e3eb5d990 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.350+0000 7fc716a00640 1 -- 172.21.15.39:0/1101541516 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc724004b80 con 0x562e3eb5d990 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.351+0000 7fc716a00640 1 -- 172.21.15.39:0/1101541516 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc72405a050 con 0x562e3eb5d990 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.351+0000 7fc73ea8b740 10 client.25853 fetch_fsmap learned FSMap version 33 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.351+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562e3ef480a0 con 0x562e3eb5d990 2024-09-21T01:11:31.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.351+0000 7fc716a00640 1 -- 172.21.15.39:0/1101541516 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72800fe50 con 0x562e3eb5d990 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.351+0000 7fc716a00640 1 -- 172.21.15.39:0/1101541516 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7fc724032540 con 0x562e3eb5d990 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc73ea8b740 10 client.25853 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc73ea8b740 10 client.25853 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc73ea8b740 20 client.25853 populate_metadata read hostname 'smithi039' 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb451d0 0x562e3ef4b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc73ea8b740 4 client.25853 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e3eea59e0 con 0x562e3eb451d0 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.352+0000 7fc734a00640 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb451d0 0x562e3ef4b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.353+0000 7fc734a00640 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb451d0 0x562e3ef4b660 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fd53a8f109b20c2 server_cookie=330e9e6bbd4bc6e4 in_seq=0 out_seq=0 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.353+0000 7fc716a00640 10 client.25853 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc716a00640 1 -- 172.21.15.39:0/1101541516 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562e3eea59e0 con 0x562e3eb451d0 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc716a00640 10 client.25853 handle_command_reply: tid=0 2024-09-21T01:11:31.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 client.25853 shutdown 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 msgr2=0x562e3ef39200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef39200 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc724009900 tx=0x7fc724009260 comp rx=0 tx=0).stop 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 shutdown_connections 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb451d0 0x562e3ef4b660 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb86850 0x562e3ef454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb65b70 0x562e3ef39780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1101541516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef39200 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc716a00640 0 client.25853 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 >> 172.21.15.39:0/1101541516 conn(0x562e3eb7ac40 msgr2=0x562e3ef30b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 shutdown_connections 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1101541516 wait complete. 2024-09-21T01:11:31.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.354+0000 7fc73ea8b740 20 client.25853 trim_cache size 0 max 16384 2024-09-21T01:11:31.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 Processor -- start 2024-09-21T01:11:31.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 -- start start 2024-09-21T01:11:31.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 0x562e3ef3b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb86850 0x562e3ef45920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc710012980 con 0x562e3eb65b70 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc7100101f0 con 0x562e3eb86850 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc71000e060 con 0x562e3eb5d990 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc734a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc735400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 0x562e3ef3b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc735400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 0x562e3ef3b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39298/0 (socket says 172.21.15.39:39298) 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.373+0000 7fc735400640 1 -- 172.21.15.39:0/757589348 learned_addr learned my addr 172.21.15.39:0/757589348 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735e00640 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb86850 0x562e3ef45920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735400640 1 -- 172.21.15.39:0/757589348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb86850 msgr2=0x562e3ef45920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735400640 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb86850 0x562e3ef45920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735400640 1 -- 172.21.15.39:0/757589348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 msgr2=0x562e3ef453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735400640 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735400640 1 -- 172.21.15.39:0/757589348 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3ef31ac0 con 0x562e3eb5d990 2024-09-21T01:11:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc735400640 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 0x562e3ef3b0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc724032b00 tx=0x7fc724004490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a8ef4107438b3d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:31.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.374+0000 7fc721c00640 1 -- 172.21.15.39:0/757589348 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724062070 con 0x562e3eb5d990 2024-09-21T01:11:31.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.375+0000 7fc721c00640 1 -- 172.21.15.39:0/757589348 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc724032d80 con 0x562e3eb5d990 2024-09-21T01:11:31.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.375+0000 7fc721c00640 1 -- 172.21.15.39:0/757589348 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72404e050 con 0x562e3eb5d990 2024-09-21T01:11:31.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.375+0000 7fc73ea8b740 1 -- 172.21.15.39:0/757589348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 msgr2=0x562e3ef3b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.375+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 0x562e3ef3b0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc724032b00 tx=0x7fc724004490 comp rx=0 tx=0).stop 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 -- 172.21.15.39:0/757589348 shutdown_connections 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb86850 0x562e3ef45920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/757589348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb5d990 0x562e3ef3b0e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 -- 172.21.15.39:0/757589348 >> 172.21.15.39:0/757589348 conn(0x562e3eb7ac40 msgr2=0x562e3ef29f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 -- 172.21.15.39:0/757589348 shutdown_connections 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 -- 172.21.15.39:0/757589348 wait complete. 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 -- start start 2024-09-21T01:11:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.376+0000 7fc73ea8b740 1 Processor -- start 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef3df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef3e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3efaf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc71000e6b0 con 0x562e3eb65b70 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc71000fb20 con 0x562e3eb5d990 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc71000ed80 con 0x562e3eb86850 2024-09-21T01:11:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc73ea8b740 10 client.0 _ll_register_callbacks cb 0x7fc724009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc735400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef3df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc734a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef3e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc735400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef3df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:55130/0 (socket says 172.21.15.39:55130) 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc735400640 1 -- 172.21.15.39:0/1501464977 learned_addr learned my addr 172.21.15.39:0/1501464977 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.377+0000 7fc735e00640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3efaf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735400640 1 -- 172.21.15.39:0/1501464977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 msgr2=0x562e3efaf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735400640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3efaf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735400640 1 -- 172.21.15.39:0/1501464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 msgr2=0x562e3ef3e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735400640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef3e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735400640 1 -- 172.21.15.39:0/1501464977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3efaf7d0 con 0x562e3eb5d990 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735e00640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3efaf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc735400640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef3df50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc724032990 tx=0x7fc724004300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f938a0d804b0d836 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc716a00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:11:31.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc716a00640 1 -- 172.21.15.39:0/1501464977 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72406b070 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc724032d80 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.378+0000 7fc716a00640 1 -- 172.21.15.39:0/1501464977 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc724004b80 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc716a00640 1 -- 172.21.15.39:0/1501464977 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc724077050 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 10 client.25859 fetch_fsmap learned FSMap version 33 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562e3efb6ba0 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc716a00640 1 -- 172.21.15.39:0/1501464977 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc724032d80 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc716a00640 1 -- 172.21.15.39:0/1501464977 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7fc72404a450 con 0x562e3eb5d990 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 10 client.25859 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 10 client.25859 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 20 client.25859 populate_metadata read hostname 'smithi039' 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb1a580 0x562e3efb9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 4 client.25859 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}] 2024-09-21T01:11:31.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.379+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}) -- 0x562e3eea1490 con 0x562e3eb1a580 2024-09-21T01:11:31.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.380+0000 7fc734a00640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb1a580 0x562e3efb9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:31.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.380+0000 7fc734a00640 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb1a580 0x562e3efb9310 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e814e502c73e3a6a server_cookie=e054c108b74ade33 in_seq=0 out_seq=0 2024-09-21T01:11:31.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.380+0000 7fc716a00640 10 client.25859 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:31.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:31 smithi039 ceph-mon[30415]: pgmap v633: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 106 op/s 2024-09-21T01:11:31.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:31 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:11:31.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:31 smithi150 ceph-mon[34945]: pgmap v633: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 106 op/s 2024-09-21T01:11:31.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:31 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:11:31.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.843+0000 7fc716a00640 1 -- 172.21.15.39:0/1501464977 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562e3eea1490 con 0x562e3eb1a580 2024-09-21T01:11:31.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.843+0000 7fc716a00640 10 client.25859 handle_command_reply: tid=0 2024-09-21T01:11:31.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.843+0000 7fc73ea8b740 1 client.25859 shutdown 2024-09-21T01:11:31.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 msgr2=0x562e3ef3df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef3df50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc724032990 tx=0x7fc724004300 comp rx=0 tx=0).stop 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 shutdown_connections 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x562e3eb1a580 0x562e3efb9310 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x562e3eb86850 0x562e3efaf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e3eb65b70 0x562e3ef3e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 --2- 172.21.15.39:0/1501464977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e3eb5d990 0x562e3ef3df50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc716a00640 0 client.25859 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 >> 172.21.15.39:0/1501464977 conn(0x562e3eb7ac40 msgr2=0x562e3ef29f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 shutdown_connections 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 1 -- 172.21.15.39:0/1501464977 wait complete. 2024-09-21T01:11:31.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.844+0000 7fc73ea8b740 20 client.25859 trim_cache size 0 max 16384 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 33, 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 30, 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "22c806d5": { 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 30, 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:11:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 141.1, 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:31.848 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7180b6680 msgr2=0x7fc7180b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7180b6680 0x7fc7180b8b90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc724005ab0 tx=0x7fc724005a20 comp rx=0 tx=0).stop 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 msgr2=0x7fc73019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 0x7fc73019cc90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00e970 tx=0x7fc71c00ee30 comp rx=0 tx=0).stop 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 shutdown_connections 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc730110150 0x7fc7301a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc730076080 0x7fc73019cc90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.845+0000 7fc737e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7180b6680 0x7fc7180b8b90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.846+0000 7fc737e00640 1 --2- 172.21.15.39:0/2450023193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc730075730 0x7fc73019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:31.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.846+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 >> 172.21.15.39:0/2450023193 conn(0x7fc7300ff5a0 msgr2=0x7fc730100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:31.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.846+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 shutdown_connections 2024-09-21T01:11:31.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:31.846+0000 7fc737e00640 1 -- 172.21.15.39:0/2450023193 wait complete. 2024-09-21T01:11:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:31 smithi160 ceph-mon[34696]: pgmap v633: 129 pgs: 129 active+clean; 230 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 106 op/s 2024-09-21T01:11:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:31 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:11:32.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.94 MB/sec execute 239 sec latency 1401.235 ms 2024-09-21T01:11:32.660 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '22c806d5', quiesce took 5.0 seconds. Will release after: 16.05 2024-09-21T01:11:33.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.92 MB/sec execute 240 sec latency 2401.298 ms 2024-09-21T01:11:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:33 smithi150 ceph-mon[34945]: pgmap v634: 129 pgs: 129 active+clean; 229 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 108 op/s 2024-09-21T01:11:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:33 smithi160 ceph-mon[34696]: pgmap v634: 129 pgs: 129 active+clean; 229 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 108 op/s 2024-09-21T01:11:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:33 smithi039 ceph-mon[30415]: pgmap v634: 129 pgs: 129 active+clean; 229 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 108 op/s 2024-09-21T01:11:34.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.90 MB/sec execute 241 sec latency 3401.388 ms 2024-09-21T01:11:35.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.88 MB/sec execute 242 sec latency 4401.547 ms 2024-09-21T01:11:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:35 smithi150 ceph-mon[34945]: pgmap v635: 129 pgs: 129 active+clean; 229 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 1009 KiB/s wr, 94 op/s 2024-09-21T01:11:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:35 smithi160 ceph-mon[34696]: pgmap v635: 129 pgs: 129 active+clean; 229 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 1009 KiB/s wr, 94 op/s 2024-09-21T01:11:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:35 smithi039 ceph-mon[30415]: pgmap v635: 129 pgs: 129 active+clean; 229 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 1009 KiB/s wr, 94 op/s 2024-09-21T01:11:36.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.86 MB/sec execute 243 sec latency 5401.706 ms 2024-09-21T01:11:37.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.84 MB/sec execute 244 sec latency 6401.844 ms 2024-09-21T01:11:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:37 smithi150 ceph-mon[34945]: pgmap v636: 129 pgs: 129 active+clean; 231 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-09-21T01:11:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:37 smithi160 ceph-mon[34696]: pgmap v636: 129 pgs: 129 active+clean; 231 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-09-21T01:11:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:37 smithi039 ceph-mon[30415]: pgmap v636: 129 pgs: 129 active+clean; 231 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-09-21T01:11:38.167 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.82 MB/sec execute 245 sec latency 7401.908 ms 2024-09-21T01:11:39.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.80 MB/sec execute 246 sec latency 8402.054 ms 2024-09-21T01:11:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:39 smithi150 ceph-mon[34945]: pgmap v637: 129 pgs: 129 active+clean; 233 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-09-21T01:11:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:39 smithi160 ceph-mon[34696]: pgmap v637: 129 pgs: 129 active+clean; 233 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-09-21T01:11:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:39 smithi039 ceph-mon[30415]: pgmap v637: 129 pgs: 129 active+clean; 233 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.2 MiB/s wr, 97 op/s 2024-09-21T01:11:40.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.78 MB/sec execute 247 sec latency 9402.224 ms 2024-09-21T01:11:41.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.76 MB/sec execute 248 sec latency 10402.447 ms 2024-09-21T01:11:41.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:41 smithi039 ceph-mon[30415]: pgmap v638: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 838 KiB/s wr, 63 op/s 2024-09-21T01:11:41.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:41 smithi150 ceph-mon[34945]: pgmap v638: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 838 KiB/s wr, 63 op/s 2024-09-21T01:11:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:41 smithi160 ceph-mon[34696]: pgmap v638: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 838 KiB/s wr, 63 op/s 2024-09-21T01:11:42.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.74 MB/sec execute 249 sec latency 11402.516 ms 2024-09-21T01:11:43.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.72 MB/sec execute 250 sec latency 12402.577 ms 2024-09-21T01:11:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:43 smithi150 ceph-mon[34945]: pgmap v639: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 724 KiB/s wr, 52 op/s 2024-09-21T01:11:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:43 smithi160 ceph-mon[34696]: pgmap v639: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 724 KiB/s wr, 52 op/s 2024-09-21T01:11:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:43 smithi039 ceph-mon[30415]: pgmap v639: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 724 KiB/s wr, 52 op/s 2024-09-21T01:11:44.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.70 MB/sec execute 251 sec latency 13402.664 ms 2024-09-21T01:11:45.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.68 MB/sec execute 252 sec latency 14402.830 ms 2024-09-21T01:11:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:45 smithi150 ceph-mon[34945]: pgmap v640: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 852 KiB/s wr, 52 op/s 2024-09-21T01:11:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:45 smithi160 ceph-mon[34696]: pgmap v640: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 852 KiB/s wr, 52 op/s 2024-09-21T01:11:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:45 smithi039 ceph-mon[30415]: pgmap v640: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 852 KiB/s wr, 52 op/s 2024-09-21T01:11:46.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.66 MB/sec execute 253 sec latency 15402.981 ms 2024-09-21T01:11:47.168 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.64 MB/sec execute 254 sec latency 16403.051 ms 2024-09-21T01:11:47.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:47 smithi039 ceph-mon[30415]: pgmap v641: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 838 KiB/s wr, 45 op/s 2024-09-21T01:11:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:47 smithi150 ceph-mon[34945]: pgmap v641: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 838 KiB/s wr, 45 op/s 2024-09-21T01:11:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:47 smithi160 ceph-mon[34696]: pgmap v641: 129 pgs: 129 active+clean; 234 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 838 KiB/s wr, 45 op/s 2024-09-21T01:11:48.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.63 MB/sec execute 255 sec latency 17403.169 ms 2024-09-21T01:11:48.711 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:11:49.101 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:11:49.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.61 MB/sec execute 256 sec latency 18403.297 ms 2024-09-21T01:11:49.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:49 smithi039 ceph-mon[30415]: pgmap v642: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 704 KiB/s wr, 37 op/s 2024-09-21T01:11:49.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.772+0000 7feaede00640 1 -- 172.21.15.39:0/2162298468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 msgr2=0x7feae8110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:49.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.772+0000 7feaede00640 1 --2- 172.21.15.39:0/2162298468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 0x7feae8110470 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feae0009ca0 tx=0x7feae002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:11:49.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 -- 172.21.15.39:0/2162298468 shutdown_connections 2024-09-21T01:11:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 --2- 172.21.15.39:0/2162298468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 0x7feae8110470 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 --2- 172.21.15.39:0/2162298468 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feae81092f0 0x7feae8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 --2- 172.21.15.39:0/2162298468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 -- 172.21.15.39:0/2162298468 >> 172.21.15.39:0/2162298468 conn(0x7feae80ff440 msgr2=0x7feae8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 -- 172.21.15.39:0/2162298468 shutdown_connections 2024-09-21T01:11:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.773+0000 7feaede00640 1 -- 172.21.15.39:0/2162298468 wait complete. 2024-09-21T01:11:49.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.774+0000 7feaede00640 1 Processor -- start 2024-09-21T01:11:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.774+0000 7feaede00640 1 -- start start 2024-09-21T01:11:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaede00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaede00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feae81092f0 0x7feae8075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaede00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 0x7feae8076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaede00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7feae8113900 con 0x7feae8109c40 2024-09-21T01:11:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaede00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feae8113780 con 0x7feae81039b0 2024-09-21T01:11:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaede00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feae8113a80 con 0x7feae81092f0 2024-09-21T01:11:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaeca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaed400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 0x7feae8076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaeca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50706/0 (socket says 172.21.15.39:50706) 2024-09-21T01:11:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feaeca00640 1 -- 172.21.15.39:0/2327348508 learned_addr learned my addr 172.21.15.39:0/2327348508 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.775+0000 7feae7e00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feae81092f0 0x7feae8075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaeca00640 1 -- 172.21.15.39:0/2327348508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feae81092f0 msgr2=0x7feae8075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaeca00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feae81092f0 0x7feae8075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaeca00640 1 -- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 msgr2=0x7feae8076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaeca00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 0x7feae8076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaeca00640 1 -- 172.21.15.39:0/2327348508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae81a8570 con 0x7feae81039b0 2024-09-21T01:11:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaeca00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8075710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feadc00d950 tx=0x7feadc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc017030 con 0x7feae81039b0 2024-09-21T01:11:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae81a8860 con 0x7feae81039b0 2024-09-21T01:11:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feadc004020 con 0x7feae81039b0 2024-09-21T01:11:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc0047a0 con 0x7feae81039b0 2024-09-21T01:11:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.776+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae81a8da0 con 0x7feae81039b0 2024-09-21T01:11:49.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.777+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae8103dc0 con 0x7feae81039b0 2024-09-21T01:11:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.781+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7feadc009a30 con 0x7feae81039b0 2024-09-21T01:11:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.782+0000 7feae5600640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fead00b6680 0x7fead00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.782+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feadc0e7bd0 con 0x7feae81039b0 2024-09-21T01:11:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.783+0000 7feae7e00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fead00b6680 0x7fead00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.783+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7feadc0adce0 con 0x7feae81039b0 2024-09-21T01:11:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:49.783+0000 7feae7e00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fead00b6680 0x7fead00b8b90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7feae8108de0 tx=0x7fead400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:49 smithi150 ceph-mon[34945]: pgmap v642: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 704 KiB/s wr, 37 op/s 2024-09-21T01:11:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:49 smithi160 ceph-mon[34696]: pgmap v642: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 704 KiB/s wr, 37 op/s 2024-09-21T01:11:50.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.022+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feae8076a10 con 0x7feae81039b0 2024-09-21T01:11:50.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.023+0000 7feae5600640 1 -- 172.21.15.39:0/2327348508 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11915 (secure 0 0 0) 0x7feadc0b20c0 con 0x7feae81039b0 2024-09-21T01:11:50.025 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:11:50.025 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":33,"btime":"2024-09-21T01:11:30:411273+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:11:29.517648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:11:50.025 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 33 2024-09-21T01:11:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.026+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fead00b6680 msgr2=0x7fead00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.026+0000 7feaede00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fead00b6680 0x7fead00b8b90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7feae8108de0 tx=0x7fead400b390 comp rx=0 tx=0).stop 2024-09-21T01:11:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 msgr2=0x7feae8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8075710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feadc00d950 tx=0x7feadc00de10 comp rx=0 tx=0).stop 2024-09-21T01:11:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 shutdown_connections 2024-09-21T01:11:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feae8109c40 0x7feae8076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:50.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feae81092f0 0x7feae8075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:50.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fead00b6680 0x7fead00b8b90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:50.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 --2- 172.21.15.39:0/2327348508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae81039b0 0x7feae8075710 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:50.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 >> 172.21.15.39:0/2327348508 conn(0x7feae80ff440 msgr2=0x7feae8101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:50.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 shutdown_connections 2024-09-21T01:11:50.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:50.027+0000 7feaede00640 1 -- 172.21.15.39:0/2327348508 wait complete. 2024-09-21T01:11:50.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.59 MB/sec execute 257 sec latency 19403.395 ms 2024-09-21T01:11:50.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:50 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2327348508' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:50.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:50 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2327348508' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:50.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:50 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2327348508' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:50.930 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 22c806d5 --release --await' 2024-09-21T01:11:50.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 22c806d5 --release --await 2024-09-21T01:11:51.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.57 MB/sec execute 258 sec latency 20403.481 ms 2024-09-21T01:11:51.310 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:11:51.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:51 smithi039 ceph-mon[30415]: pgmap v643: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 540 KiB/s wr, 33 op/s 2024-09-21T01:11:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:51 smithi150 ceph-mon[34945]: pgmap v643: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 540 KiB/s wr, 33 op/s 2024-09-21T01:11:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:51 smithi160 ceph-mon[34696]: pgmap v643: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 540 KiB/s wr, 33 op/s 2024-09-21T01:11:51.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 -- 172.21.15.39:0/1295556613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010b6c0 msgr2=0x7fd55010bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 --2- 172.21.15.39:0/1295556613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010b6c0 0x7fd55010bad0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009900 tx=0x7fd53c02a420 comp rx=0 tx=0).stop 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 -- 172.21.15.39:0/1295556613 shutdown_connections 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 --2- 172.21.15.39:0/1295556613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd55010c010 0x7fd550112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 --2- 172.21.15.39:0/1295556613 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010b6c0 0x7fd55010bad0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 --2- 172.21.15.39:0/1295556613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd550105d80 0x7fd550106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 -- 172.21.15.39:0/1295556613 >> 172.21.15.39:0/1295556613 conn(0x7fd550101810 msgr2=0x7fd550103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 -- 172.21.15.39:0/1295556613 shutdown_connections 2024-09-21T01:11:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.982+0000 7fd555800640 1 -- 172.21.15.39:0/1295556613 wait complete. 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 Processor -- start 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 -- start start 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 0x7fd55010fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55010b6c0 0x7fd550110240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010c010 0x7fd550110780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd550115d60 con 0x7fd550105d80 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd550115be0 con 0x7fd55010b6c0 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.984+0000 7fd555800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd550115ee0 con 0x7fd55010c010 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 0x7fd55010fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 0x7fd55010fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48516/0 (socket says 172.21.15.39:48516) 2024-09-21T01:11:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 -- 172.21.15.39:0/354123899 learned_addr learned my addr 172.21.15.39:0/354123899 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54e000640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55010b6c0 0x7fd550110240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 -- 172.21.15.39:0/354123899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010c010 msgr2=0x7fd550110780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:11:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010c010 0x7fd550110780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 -- 172.21.15.39:0/354123899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55010b6c0 msgr2=0x7fd550110240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55010b6c0 0x7fd550110240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.985+0000 7fd54ea00640 1 -- 172.21.15.39:0/354123899 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5501acc30 con 0x7fd550105d80 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.986+0000 7fd54ea00640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 0x7fd55010fd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd54000b450 tx=0x7fd54000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.986+0000 7fd547400640 1 -- 172.21.15.39:0/354123899 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd540015020 con 0x7fd550105d80 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.986+0000 7fd547400640 1 -- 172.21.15.39:0/354123899 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd540011d40 con 0x7fd550105d80 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.986+0000 7fd547400640 1 -- 172.21.15.39:0/354123899 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54000f340 con 0x7fd550105d80 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.986+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5501acf20 con 0x7fd550105d80 2024-09-21T01:11:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.987+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5501ad400 con 0x7fd550105d80 2024-09-21T01:11:51.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.990+0000 7fd55c47d740 1 Processor -- start 2024-09-21T01:11:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 -- start start 2024-09-21T01:11:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d41f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d41f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d41fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55794d396290 con 0x55794d078520 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55794d3856c0 con 0x55794d057840 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd55c47d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55794d3894d0 con 0x55794d04f660 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd54f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d41f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd54f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d41f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48530/0 (socket says 172.21.15.39:48530) 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd54f400640 1 -- 172.21.15.39:0/1589843195 learned_addr learned my addr 172.21.15.39:0/1589843195 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.991+0000 7fd547400640 1 -- 172.21.15.39:0/354123899 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fd540018020 con 0x7fd550105d80 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54e000640 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d41fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54ea00640 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d41f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:51.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54f400640 1 -- 172.21.15.39:0/1589843195 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 msgr2=0x55794d41f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:51.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54f400640 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d41f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54f400640 1 -- 172.21.15.39:0/1589843195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 msgr2=0x55794d41fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:51.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54f400640 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d41fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.992+0000 7fd54f400640 1 -- 172.21.15.39:0/1589843195 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794d312460 con 0x55794d078520 2024-09-21T01:11:51.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.993+0000 7fd54f400640 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d41f1c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd548009e30 tx=0x7fd54800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f368b3a4f06938 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:51.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.993+0000 7fd53be00640 1 -- 172.21.15.39:0/1589843195 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548003b70 con 0x55794d078520 2024-09-21T01:11:51.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.993+0000 7fd53be00640 1 -- 172.21.15.39:0/1589843195 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd54800ee70 con 0x55794d078520 2024-09-21T01:11:51.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.993+0000 7fd53be00640 1 -- 172.21.15.39:0/1589843195 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5480106d0 con 0x55794d078520 2024-09-21T01:11:51.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.994+0000 7fd547400640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5340b6780 0x7fd5340b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.994+0000 7fd547400640 1 -- 172.21.15.39:0/354123899 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd5400e7210 con 0x7fd550105d80 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.994+0000 7fd55c47d740 1 -- 172.21.15.39:0/1589843195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 msgr2=0x55794d41f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.994+0000 7fd55c47d740 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d41f1c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd548009e30 tx=0x7fd54800e720 comp rx=0 tx=0).stop 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.994+0000 7fd54ea00640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5340b6780 0x7fd5340b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 -- 172.21.15.39:0/1589843195 shutdown_connections 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d41fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d41f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 --2- 172.21.15.39:0/1589843195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d41f1c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 -- 172.21.15.39:0/1589843195 >> 172.21.15.39:0/1589843195 conn(0x55794d06c910 msgr2=0x55794d41d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:51.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 -- 172.21.15.39:0/1589843195 shutdown_connections 2024-09-21T01:11:51.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.995+0000 7fd55c47d740 1 -- 172.21.15.39:0/1589843195 wait complete. 2024-09-21T01:11:51.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.996+0000 7fd54ea00640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5340b6780 0x7fd5340b8c90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55794d41de40 tx=0x7fd5400ea7a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:51.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.996+0000 7fd55c47d740 1 -- start start 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.996+0000 7fd55c47d740 1 Processor -- start 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d04f660 0x55794d42ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 0x55794d42b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d078520 0x55794d436cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55794d1e6bb0 con 0x55794d04f660 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55794d3206c0 con 0x55794d078520 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55794d42d9a0 con 0x55794d057840 2024-09-21T01:11:51.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd55c47d740 10 client.0 _ll_register_callbacks cb 0x55794d33b490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd54e000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 0x55794d42b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd54e000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 0x55794d42b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:60678/0 (socket says 172.21.15.39:60678) 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd54e000640 1 -- 172.21.15.39:0/40977275 learned_addr learned my addr 172.21.15.39:0/40977275 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.997+0000 7fd54f400640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d04f660 0x55794d42ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54f400640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d078520 0x55794d436cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54e000640 1 -- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d078520 msgr2=0x55794d436cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54e000640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d078520 0x55794d436cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54e000640 1 -- 172.21.15.39:0/40977275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d04f660 msgr2=0x55794d42ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54e000640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d04f660 0x55794d42ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54e000640 1 -- 172.21.15.39:0/40977275 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794d0e3b00 con 0x55794d057840 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.998+0000 7fd54f400640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d04f660 0x55794d42ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd54e000640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 0x55794d42b320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0096c0 tx=0x7fd53c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7cc1568aff60723 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd538c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd538c00640 1 -- 172.21.15.39:0/40977275 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c035040 con 0x55794d057840 2024-09-21T01:11:52.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd54800ee70 con 0x55794d057840 2024-09-21T01:11:52.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd538c00640 1 -- 172.21.15.39:0/40977275 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd53c0048c0 con 0x55794d057840 2024-09-21T01:11:52.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd538c00640 1 -- 172.21.15.39:0/40977275 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd53c004ac0 con 0x55794d057840 2024-09-21T01:11:52.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd55c47d740 10 client.25864 fetch_fsmap learned FSMap version 33 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:51.999+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55794d15a960 con 0x55794d057840 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd538c00640 1 -- 172.21.15.39:0/40977275 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd54800ee70 con 0x55794d057840 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd538c00640 1 -- 172.21.15.39:0/40977275 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7fd53c0314a0 con 0x55794d057840 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd55c47d740 10 client.25864 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd55c47d740 10 client.25864 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd55c47d740 20 client.25864 populate_metadata read hostname 'smithi039' 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd55c47d740 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d036ea0 0x55794d43cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd55c47d740 4 client.25864 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.000+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55794d3981e0 con 0x55794d036ea0 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.001+0000 7fd54f400640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d036ea0 0x55794d43cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.001+0000 7fd54f400640 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d036ea0 0x55794d43cd00 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68ddfed8699627e5 server_cookie=41085c3a139fad5 in_seq=0 out_seq=0 2024-09-21T01:11:52.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.001+0000 7fd538c00640 10 client.25864 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:52.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd538c00640 1 -- 172.21.15.39:0/40977275 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55794d3981e0 con 0x55794d036ea0 2024-09-21T01:11:52.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd538c00640 10 client.25864 handle_command_reply: tid=0 2024-09-21T01:11:52.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 client.25864 shutdown 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 msgr2=0x55794d42b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 0x55794d42b320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0096c0 tx=0x7fd53c005270 comp rx=0 tx=0).stop 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 shutdown_connections 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d036ea0 0x55794d43cd00 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d078520 0x55794d436cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d057840 0x55794d42b320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 --2- 172.21.15.39:0/40977275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d04f660 0x55794d42ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.002+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 >> 172.21.15.39:0/40977275 conn(0x55794d06c910 msgr2=0x55794d422370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.003+0000 7fd538c00640 0 client.25864 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.003+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 shutdown_connections 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.003+0000 7fd55c47d740 1 -- 172.21.15.39:0/40977275 wait complete. 2024-09-21T01:11:52.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.003+0000 7fd55c47d740 20 client.25864 trim_cache size 0 max 16384 2024-09-21T01:11:52.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.021+0000 7fd55c47d740 1 Processor -- start 2024-09-21T01:11:52.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.021+0000 7fd55c47d740 1 -- start start 2024-09-21T01:11:52.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.022+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d04f660 0x55794d425090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.022+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d057840 0x55794d429460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.022+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 0x55794d4299a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.022+0000 7fd55c47d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd5501ba070 con 0x55794d057840 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.022+0000 7fd55c47d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5501b78e0 con 0x55794d04f660 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.022+0000 7fd55c47d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd5501b5750 con 0x55794d078520 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 0x55794d4299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 0x55794d4299a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:60692/0 (socket says 172.21.15.39:60692) 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 -- 172.21.15.39:0/2793925352 learned_addr learned my addr 172.21.15.39:0/2793925352 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54e000640 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d057840 0x55794d429460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d04f660 0x55794d425090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 -- 172.21.15.39:0/2793925352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d04f660 msgr2=0x55794d425090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d04f660 0x55794d425090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 -- 172.21.15.39:0/2793925352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d057840 msgr2=0x55794d429460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d057840 0x55794d429460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 -- 172.21.15.39:0/2793925352 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794d4214b0 con 0x55794d078520 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54e000640 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d057840 0x55794d429460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.023+0000 7fd54f400640 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 0x55794d4299a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd548009270 tx=0x7fd548009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7014baf901e7ebb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:52.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.024+0000 7fd53b400640 1 -- 172.21.15.39:0/2793925352 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548004b20 con 0x55794d078520 2024-09-21T01:11:52.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.024+0000 7fd53b400640 1 -- 172.21.15.39:0/2793925352 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd548004cc0 con 0x55794d078520 2024-09-21T01:11:52.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.024+0000 7fd53b400640 1 -- 172.21.15.39:0/2793925352 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5480053a0 con 0x55794d078520 2024-09-21T01:11:52.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 -- 172.21.15.39:0/2793925352 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 msgr2=0x55794d4299a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:52.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 0x55794d4299a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd548009270 tx=0x7fd548009500 comp rx=0 tx=0).stop 2024-09-21T01:11:52.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 -- 172.21.15.39:0/2793925352 shutdown_connections 2024-09-21T01:11:52.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d078520 0x55794d4299a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d057840 0x55794d429460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 --2- 172.21.15.39:0/2793925352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d04f660 0x55794d425090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.025+0000 7fd55c47d740 1 -- 172.21.15.39:0/2793925352 >> 172.21.15.39:0/2793925352 conn(0x55794d06c910 msgr2=0x55794d42ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:52.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.028+0000 7fd55c47d740 1 -- 172.21.15.39:0/2793925352 shutdown_connections 2024-09-21T01:11:52.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.028+0000 7fd55c47d740 1 -- 172.21.15.39:0/2793925352 wait complete. 2024-09-21T01:11:52.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.028+0000 7fd55c47d740 1 -- start start 2024-09-21T01:11:52.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 Processor -- start 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d4335e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d427cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d4283e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd5501b5da0 con 0x55794d078520 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5501b7210 con 0x55794d057840 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd5501b6470 con 0x55794d04f660 2024-09-21T01:11:52.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.029+0000 7fd55c47d740 10 client.0 _ll_register_callbacks cb 0x7fd53c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:11:52.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d427cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d427cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50766/0 (socket says 172.21.15.39:50766) 2024-09-21T01:11:52.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 -- 172.21.15.39:0/3644594291 learned_addr learned my addr 172.21.15.39:0/3644594291 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 -- 172.21.15.39:0/3644594291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 msgr2=0x55794d4335e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d4335e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 -- 172.21.15.39:0/3644594291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 msgr2=0x55794d4283e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d4283e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.030+0000 7fd54e000640 1 -- 172.21.15.39:0/3644594291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794d432220 con 0x55794d057840 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.031+0000 7fd54e000640 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d427cd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd53c02a460 tx=0x7fd53c031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d43d3fc0b2e286 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.031+0000 7fd538c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.031+0000 7fd538c00640 1 -- 172.21.15.39:0/3644594291 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c056030 con 0x55794d057840 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.031+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd548004cc0 con 0x55794d057840 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.031+0000 7fd538c00640 1 -- 172.21.15.39:0/3644594291 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd53c0046c0 con 0x55794d057840 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.031+0000 7fd538c00640 1 -- 172.21.15.39:0/3644594291 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd53c060050 con 0x55794d057840 2024-09-21T01:11:52.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.032+0000 7fd55c47d740 10 client.25880 fetch_fsmap learned FSMap version 33 2024-09-21T01:11:52.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.032+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55794d4a78e0 con 0x55794d057840 2024-09-21T01:11:52.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.032+0000 7fd538c00640 1 -- 172.21.15.39:0/3644594291 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd548004cc0 con 0x55794d057840 2024-09-21T01:11:52.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.032+0000 7fd538c00640 1 -- 172.21.15.39:0/3644594291 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7fd53c003aa0 con 0x55794d057840 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd55c47d740 10 client.25880 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd55c47d740 10 client.25880 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd55c47d740 20 client.25880 populate_metadata read hostname 'smithi039' 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd55c47d740 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d00c250 0x55794d4aa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd55c47d740 4 client.25880 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "22c806d5", "release": true, "await": true}] 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "22c806d5", "release": true, "await": true}) -- 0x55794d393c90 con 0x55794d00c250 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.033+0000 7fd54f400640 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d00c250 0x55794d4aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:52.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.034+0000 7fd54f400640 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d00c250 0x55794d4aa150 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77fd32994016f032 server_cookie=ae0f4c8e249b8ac7 in_seq=0 out_seq=0 2024-09-21T01:11:52.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.034+0000 7fd538c00640 10 client.25880 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:52.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd538c00640 1 -- 172.21.15.39:0/3644594291 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55794d393c90 con 0x55794d00c250 2024-09-21T01:11:52.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd538c00640 10 client.25880 handle_command_reply: tid=0 2024-09-21T01:11:52.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 client.25880 shutdown 2024-09-21T01:11:52.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 msgr2=0x55794d427cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:52.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d427cd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd53c02a460 tx=0x7fd53c031750 comp rx=0 tx=0).stop 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 shutdown_connections 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55794d00c250 0x55794d4aa150 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55794d078520 0x55794d4283e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55794d057840 0x55794d427cd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd538c00640 0 client.25880 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 --2- 172.21.15.39:0/3644594291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55794d04f660 0x55794d4335e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 >> 172.21.15.39:0/3644594291 conn(0x55794d06c910 msgr2=0x55794d4293e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.039+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 shutdown_connections 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.040+0000 7fd55c47d740 1 -- 172.21.15.39:0/3644594291 wait complete. 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.040+0000 7fd55c47d740 20 client.25880 trim_cache size 0 max 16384 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 33, 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 32, 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:11:52.042 INFO:teuthology.orchestra.run.smithi039.stdout: "22c806d5": { 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 32, 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 141.1, 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:11:52.043 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5340b6780 msgr2=0x7fd5340b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5340b6780 0x7fd5340b8c90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55794d41de40 tx=0x7fd5400ea7a0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 msgr2=0x7fd55010fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:52.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 0x7fd55010fd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd54000b450 tx=0x7fd54000b910 comp rx=0 tx=0).stop 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 shutdown_connections 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd55010c010 0x7fd550110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fd5340b6780 0x7fd5340b8c90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55010b6c0 0x7fd550110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 --2- 172.21.15.39:0/354123899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd550105d80 0x7fd55010fd00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.041+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 >> 172.21.15.39:0/354123899 conn(0x7fd550101810 msgr2=0x7fd550103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.042+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 shutdown_connections 2024-09-21T01:11:52.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:52.042+0000 7fd555800640 1 -- 172.21.15.39:0/354123899 wait complete. 2024-09-21T01:11:52.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 358851 4.56 MB/sec execute 259 sec latency 21403.528 ms 2024-09-21T01:11:53.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 359737 4.56 MB/sec execute 260 sec latency 21704.184 ms 2024-09-21T01:11:53.361 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '22c806d5', total seconds elapsed: 25.7 2024-09-21T01:11:53.361 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.05 2024-09-21T01:11:53.361 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:11:53.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:53 smithi039 ceph-mon[30415]: pgmap v644: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 562 KiB/s wr, 33 op/s 2024-09-21T01:11:53.755 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:11:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:53 smithi150 ceph-mon[34945]: pgmap v644: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 562 KiB/s wr, 33 op/s 2024-09-21T01:11:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:53 smithi160 ceph-mon[34696]: pgmap v644: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 562 KiB/s wr, 33 op/s 2024-09-21T01:11:54.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 359941 4.54 MB/sec execute 261 sec latency 810.709 ms 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 -- 172.21.15.39:0/2537449973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 msgr2=0x7f61281098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 --2- 172.21.15.39:0/2537449973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f61281098b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6110009900 tx=0x7f611002a410 comp rx=0 tx=0).stop 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 -- 172.21.15.39:0/2537449973 shutdown_connections 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 --2- 172.21.15.39:0/2537449973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6128109df0 0x7f6128110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 --2- 172.21.15.39:0/2537449973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f61281098b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 --2- 172.21.15.39:0/2537449973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6128103b60 0x7f6128103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 -- 172.21.15.39:0/2537449973 >> 172.21.15.39:0/2537449973 conn(0x7f612807b850 msgr2=0x7f612807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 -- 172.21.15.39:0/2537449973 shutdown_connections 2024-09-21T01:11:54.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.460+0000 7f612d200640 1 -- 172.21.15.39:0/2537449973 wait complete. 2024-09-21T01:11:54.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.462+0000 7f612d200640 1 Processor -- start 2024-09-21T01:11:54.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.462+0000 7f612d200640 1 -- start start 2024-09-21T01:11:54.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.462+0000 7f612d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6128103b60 0x7f612819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:54.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f612d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f612819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:54.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f612d200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6128109df0 0x7f612819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:54.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f612d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6128113ab0 con 0x7f61281094a0 2024-09-21T01:11:54.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f612d200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6128113930 con 0x7f6128109df0 2024-09-21T01:11:54.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f612d200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6128113c30 con 0x7f6128103b60 2024-09-21T01:11:54.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f612819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:54.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f612819aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34620/0 (socket says 172.21.15.39:34620) 2024-09-21T01:11:54.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 -- 172.21.15.39:0/2342442567 learned_addr learned my addr 172.21.15.39:0/2342442567 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:54.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f6125e00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6128103b60 0x7f612819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:54.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f6126800640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6128109df0 0x7f612819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 -- 172.21.15.39:0/2342442567 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6128103b60 msgr2=0x7f612819a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6128103b60 0x7f612819a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 -- 172.21.15.39:0/2342442567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6128109df0 msgr2=0x7f612819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6128109df0 0x7f612819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.463+0000 7f611fe00640 1 -- 172.21.15.39:0/2342442567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61281a03f0 con 0x7f61281094a0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.464+0000 7f611fe00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f612819aa20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6110006010 tx=0x7f6110004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.464+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611003f030 con 0x7f61281094a0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.464+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6110003aa0 con 0x7f61281094a0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.464+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61281a0680 con 0x7f61281094a0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.464+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110035070 con 0x7f61281094a0 2024-09-21T01:11:54.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.464+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61281a6e40 con 0x7f61281094a0 2024-09-21T01:11:54.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.465+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6128103f70 con 0x7f61281094a0 2024-09-21T01:11:54.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.468+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6110042030 con 0x7f61281094a0 2024-09-21T01:11:54.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.469+0000 7f611ea00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f610c0b6680 0x7f610c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:54.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.469+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f61101083b0 con 0x7f61281094a0 2024-09-21T01:11:54.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.470+0000 7f6125e00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f610c0b6680 0x7f610c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:54.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.470+0000 7f6125e00640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f610c0b6680 0x7f610c0b8b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6118005d00 tx=0x7f611800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:54.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.472+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f61100d28a0 con 0x7f61281094a0 2024-09-21T01:11:54.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.739+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f612810bbe0 con 0x7f61281094a0 2024-09-21T01:11:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.740+0000 7f611ea00640 1 -- 172.21.15.39:0/2342442567 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11915 (secure 0 0 0) 0x7f61100d77e0 con 0x7f61281094a0 2024-09-21T01:11:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:11:54.741 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":33,"btime":"2024-09-21T01:11:30:411273+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:11:29.517648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:11:54.741 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 33 2024-09-21T01:11:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f610c0b6680 msgr2=0x7f610c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f610c0b6680 0x7f610c0b8b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6118005d00 tx=0x7f611800b410 comp rx=0 tx=0).stop 2024-09-21T01:11:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 msgr2=0x7f612819aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f612819aa20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6110006010 tx=0x7f6110004060 comp rx=0 tx=0).stop 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 shutdown_connections 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f610c0b6680 0x7f610c0b8b90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6128109df0 0x7f612819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61281094a0 0x7f612819aa20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 --2- 172.21.15.39:0/2342442567 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6128103b60 0x7f612819a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 >> 172.21.15.39:0/2342442567 conn(0x7f612807b850 msgr2=0x7f6128107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.743+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 shutdown_connections 2024-09-21T01:11:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:54.744+0000 7f612d200640 1 -- 172.21.15.39:0/2342442567 wait complete. 2024-09-21T01:11:55.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 361455 4.55 MB/sec execute 262 sec latency 1173.469 ms 2024-09-21T01:11:55.592 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0' 2024-09-21T01:11:55.593 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0 2024-09-21T01:11:55.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:55 smithi150 ceph-mon[34945]: pgmap v645: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 712 KiB/s wr, 36 op/s 2024-09-21T01:11:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:55 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2342442567' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:55 smithi160 ceph-mon[34696]: pgmap v645: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 712 KiB/s wr, 36 op/s 2024-09-21T01:11:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:55 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2342442567' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:55 smithi039 ceph-mon[30415]: pgmap v645: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 712 KiB/s wr, 36 op/s 2024-09-21T01:11:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:55 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2342442567' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:11:56.008 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:11:56.169 INFO:tasks.workunit.client.0.smithi039.stdout: 1 362665 4.56 MB/sec execute 263 sec latency 18.956 ms 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 -- 172.21.15.39:0/2333356780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410be80 msgr2=0x7f8784116ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 --2- 172.21.15.39:0/2333356780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410be80 0x7f8784116ea0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f877c009860 tx=0x7f877c02a340 comp rx=0 tx=0).stop 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 -- 172.21.15.39:0/2333356780 shutdown_connections 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 --2- 172.21.15.39:0/2333356780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410be80 0x7f8784116ea0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 --2- 172.21.15.39:0/2333356780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f878410b530 0x7f878410b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 --2- 172.21.15.39:0/2333356780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8784105bf0 0x7f8784106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 -- 172.21.15.39:0/2333356780 >> 172.21.15.39:0/2333356780 conn(0x7f8784101680 msgr2=0x7f8784103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:56.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 -- 172.21.15.39:0/2333356780 shutdown_connections 2024-09-21T01:11:56.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.735+0000 7f8789400640 1 -- 172.21.15.39:0/2333356780 wait complete. 2024-09-21T01:11:56.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.737+0000 7f8789400640 1 Processor -- start 2024-09-21T01:11:56.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.737+0000 7f8789400640 1 -- start start 2024-09-21T01:11:56.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8789400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8784105bf0 0x7f87841a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8789400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 0x7f87841a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8789400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878410be80 0x7f87841a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8789400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8784119ed0 con 0x7f8784105bf0 2024-09-21T01:11:56.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8789400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8784119d50 con 0x7f878410be80 2024-09-21T01:11:56.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8789400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f878411a050 con 0x7f878410b530 2024-09-21T01:11:56.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 0x7f87841a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8783e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8784105bf0 0x7f87841a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 0x7f87841a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39794/0 (socket says 172.21.15.39:39794) 2024-09-21T01:11:56.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 -- 172.21.15.39:0/4277383793 learned_addr learned my addr 172.21.15.39:0/4277383793 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:56.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f8788a00640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878410be80 0x7f87841a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 -- 172.21.15.39:0/4277383793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878410be80 msgr2=0x7f87841a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878410be80 0x7f87841a60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 -- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8784105bf0 msgr2=0x7f87841a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8784105bf0 0x7f87841a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.738+0000 7f877be00640 1 -- 172.21.15.39:0/4277383793 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87841a67d0 con 0x7f878410b530 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.739+0000 7f877be00640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 0x7f87841a0e00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f876c00d950 tx=0x7f876c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.739+0000 7f8781600640 1 -- 172.21.15.39:0/4277383793 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c017070 con 0x7f878410b530 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.739+0000 7f8781600640 1 -- 172.21.15.39:0/4277383793 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f876c009c30 con 0x7f878410b530 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.739+0000 7f8781600640 1 -- 172.21.15.39:0/4277383793 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c003cc0 con 0x7f878410b530 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.739+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87841a6ac0 con 0x7f878410b530 2024-09-21T01:11:56.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.739+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841ad220 con 0x7f878410b530 2024-09-21T01:11:56.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 Processor -- start 2024-09-21T01:11:56.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 -- start start 2024-09-21T01:11:56.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f8781600640 1 -- 172.21.15.39:0/4277383793 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f876c003e60 con 0x7f878410b530 2024-09-21T01:11:56.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd86b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd87610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555d3fcfca90 con 0x555d3f9b6990 2024-09-21T01:11:56.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555d3fcebec0 con 0x555d3f9df850 2024-09-21T01:11:56.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.743+0000 7f87916c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555d3fcefcd0 con 0x555d3f9beb70 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd870d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34654/0 (socket says 172.21.15.39:34654) 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 -- 172.21.15.39:0/2665282996 learned_addr learned my addr 172.21.15.39:0/2665282996 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8788a00640 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd86b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f877be00640 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd87610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 -- 172.21.15.39:0/2665282996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 msgr2=0x555d3fd87610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd87610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 -- 172.21.15.39:0/2665282996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 msgr2=0x555d3fd86b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd86b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.744+0000 7f8783e00640 1 -- 172.21.15.39:0/2665282996 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3fc919d0 con 0x555d3f9b6990 2024-09-21T01:11:56.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8783e00640 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd870d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f877400af40 tx=0x7f877400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ff275a3ae52392 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:56.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8779200640 1 -- 172.21.15.39:0/2665282996 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877401d070 con 0x555d3f9b6990 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8781600640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f87680b66f0 0x7f87680b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8779200640 1 -- 172.21.15.39:0/2665282996 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8774010db0 con 0x555d3f9b6990 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8781600640 1 -- 172.21.15.39:0/4277383793 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f876c0e7610 con 0x7f878410b530 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8779200640 1 -- 172.21.15.39:0/2665282996 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774018da0 con 0x555d3f9b6990 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.745+0000 7f8788a00640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f87680b66f0 0x7f87680b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 -- 172.21.15.39:0/2665282996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 msgr2=0x555d3fd870d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd870d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f877400af40 tx=0x7f877400f850 comp rx=0 tx=0).stop 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f8788a00640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f87680b66f0 0x7f87680b8c00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f877c0303f0 tx=0x7f877c02c040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 -- 172.21.15.39:0/2665282996 shutdown_connections 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd87610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd870d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 --2- 172.21.15.39:0/2665282996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd86b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 -- 172.21.15.39:0/2665282996 >> 172.21.15.39:0/2665282996 conn(0x555d3f9d3c40 msgr2=0x555d3fd84a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:56.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 -- 172.21.15.39:0/2665282996 shutdown_connections 2024-09-21T01:11:56.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.746+0000 7f87916c1740 1 -- 172.21.15.39:0/2665282996 wait complete. 2024-09-21T01:11:56.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.747+0000 7f87916c1740 1 -- start start 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.747+0000 7f87916c1740 1 Processor -- start 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9b6990 0x555d3fd923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9beb70 0x555d3fd92930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd9e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555d3fb4d7b0 con 0x555d3f9beb70 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555d3fc90650 con 0x555d3f9df850 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x555d3fd953c0 con 0x555d3f9b6990 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f87916c1740 10 client.0 _ll_register_callbacks cb 0x555d3f9f70a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f8783e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9b6990 0x555d3fd923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f8783e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9b6990 0x555d3fd923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39816/0 (socket says 172.21.15.39:39816) 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f8783e00640 1 -- 172.21.15.39:0/4233074686 learned_addr learned my addr 172.21.15.39:0/4233074686 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f877be00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9beb70 0x555d3fd92930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.748+0000 7f8788a00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd9e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f877be00640 1 -- 172.21.15.39:0/4233074686 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9b6990 msgr2=0x555d3fd923d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f877be00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9b6990 0x555d3fd923d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f877be00640 1 -- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 msgr2=0x555d3fd9e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f877be00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd9e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f877be00640 1 -- 172.21.15.39:0/4233074686 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3fa59ba0 con 0x555d3f9beb70 2024-09-21T01:11:56.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f8788a00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd9e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f877be00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9beb70 0x555d3fd92930 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f876c0e4920 tx=0x7f876c0e4e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcee54f2d2605a5a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.749+0000 7f8771600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f8771600640 1 -- 172.21.15.39:0/4233074686 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0a8ec0 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f8771600640 1 -- 172.21.15.39:0/4233074686 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f876c0e7370 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8774010db0 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f8771600640 1 -- 172.21.15.39:0/4233074686 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f876c0ac790 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f87916c1740 10 client.15867 fetch_fsmap learned FSMap version 33 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555d3fda12c0 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f8771600640 1 -- 172.21.15.39:0/4233074686 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8774010db0 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.750+0000 7f8771600640 1 -- 172.21.15.39:0/4233074686 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7f876c098190 con 0x555d3f9beb70 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f87916c1740 10 client.15867 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f87916c1740 10 client.15867 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f87916c1740 20 client.15867 populate_metadata read hostname 'smithi039' 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f87916c1740 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f99e1d0 0x555d3fda4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f87916c1740 4 client.15867 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:11:56.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d3fcfe9e0 con 0x555d3f99e1d0 2024-09-21T01:11:56.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.751+0000 7f8783e00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f99e1d0 0x555d3fda4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.752+0000 7f8783e00640 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f99e1d0 0x555d3fda4810 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79b6ad65298ad4ef server_cookie=124d3e1ad11c193b in_seq=0 out_seq=0 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.752+0000 7f8771600640 10 client.15867 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.752+0000 7f8771600640 1 -- 172.21.15.39:0/4233074686 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555d3fcfe9e0 con 0x555d3f99e1d0 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.752+0000 7f8771600640 10 client.15867 handle_command_reply: tid=0 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.752+0000 7f87916c1740 1 client.15867 shutdown 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9beb70 msgr2=0x555d3fd92930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9beb70 0x555d3fd92930 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f876c0e4920 tx=0x7f876c0e4e90 comp rx=0 tx=0).stop 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 shutdown_connections 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f99e1d0 0x555d3fda4810 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fd9e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9beb70 0x555d3fd92930 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 --2- 172.21.15.39:0/4233074686 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9b6990 0x555d3fd923d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f8771600640 0 client.15867 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:56.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 >> 172.21.15.39:0/4233074686 conn(0x555d3f9d3c40 msgr2=0x555d3fd88580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:56.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 shutdown_connections 2024-09-21T01:11:56.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 1 -- 172.21.15.39:0/4233074686 wait complete. 2024-09-21T01:11:56.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.753+0000 7f87916c1740 20 client.15867 trim_cache size 0 max 16384 2024-09-21T01:11:56.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 Processor -- start 2024-09-21T01:11:56.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 -- start start 2024-09-21T01:11:56.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd92890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 0x555d3fd9e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9df850 0x555d3fd9eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f87841b9d60 con 0x555d3f9b6990 2024-09-21T01:11:56.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87841b75d0 con 0x555d3f9beb70 2024-09-21T01:11:56.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.772+0000 7f87916c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f87841b5440 con 0x555d3f9df850 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 0x555d3fd9e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f8783e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd92890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 0x555d3fd9e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:58446/0 (socket says 172.21.15.39:58446) 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 -- 172.21.15.39:0/1506313442 learned_addr learned my addr 172.21.15.39:0/1506313442 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f8788a00640 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9df850 0x555d3fd9eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 -- 172.21.15.39:0/1506313442 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9df850 msgr2=0x555d3fd9eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9df850 0x555d3fd9eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 -- 172.21.15.39:0/1506313442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 msgr2=0x555d3fd92890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd92890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.773+0000 7f877be00640 1 -- 172.21.15.39:0/1506313442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3fd88e80 con 0x555d3f9beb70 2024-09-21T01:11:56.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.774+0000 7f877be00640 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 0x555d3fd9e610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f876c005fd0 tx=0x7f876c0e78d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcbd891acbf5e5b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:56.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.774+0000 7f8773e00640 1 -- 172.21.15.39:0/1506313442 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c014020 con 0x555d3f9beb70 2024-09-21T01:11:56.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.774+0000 7f8773e00640 1 -- 172.21.15.39:0/1506313442 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f876c0acb30 con 0x555d3f9beb70 2024-09-21T01:11:56.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.774+0000 7f8773e00640 1 -- 172.21.15.39:0/1506313442 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0ac970 con 0x555d3f9beb70 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 -- 172.21.15.39:0/1506313442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 msgr2=0x555d3fd9e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 0x555d3fd9e610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f876c005fd0 tx=0x7f876c0e78d0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 -- 172.21.15.39:0/1506313442 shutdown_connections 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9df850 0x555d3fd9eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9beb70 0x555d3fd9e610 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 --2- 172.21.15.39:0/1506313442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd92890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 -- 172.21.15.39:0/1506313442 >> 172.21.15.39:0/1506313442 conn(0x555d3f9d3c40 msgr2=0x555d3fd92180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 -- 172.21.15.39:0/1506313442 shutdown_connections 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.775+0000 7f87916c1740 1 -- 172.21.15.39:0/1506313442 wait complete. 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.776+0000 7f87916c1740 1 -- start start 2024-09-21T01:11:56.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.776+0000 7f87916c1740 1 Processor -- start 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.776+0000 7f87916c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd96e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.776+0000 7f87916c1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd97390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.776+0000 7f87916c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fe0e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.776+0000 7f87916c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f87841b5a90 con 0x555d3f9b6990 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f87916c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87841b6f00 con 0x555d3f9df850 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f87916c1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f87841b6160 con 0x555d3f9beb70 2024-09-21T01:11:56.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f87916c1740 10 client.0 _ll_register_callbacks cb 0x7f876c0a94c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:11:56.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f8783e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd96e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f8783e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd96e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34682/0 (socket says 172.21.15.39:34682) 2024-09-21T01:11:56.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f8783e00640 1 -- 172.21.15.39:0/2963274554 learned_addr learned my addr 172.21.15.39:0/2963274554 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:11:56.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f877be00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd97390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f8788a00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fe0e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f877be00640 1 -- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 msgr2=0x555d3fe0e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f877be00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fe0e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f877be00640 1 -- 172.21.15.39:0/2963274554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 msgr2=0x555d3fd96e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:11:56.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f877be00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd96e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f877be00640 1 -- 172.21.15.39:0/2963274554 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3fe0e790 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.777+0000 7f8783e00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd96e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f877be00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd97390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f876c0aa290 tx=0x7f876c0b1b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad5a822feccaf0e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f8771600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f8771600640 1 -- 172.21.15.39:0/2963274554 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c014070 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f8771600640 1 -- 172.21.15.39:0/2963274554 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f876c0a8970 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f876c0acb30 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f8771600640 1 -- 172.21.15.39:0/2963274554 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f876c0acdf0 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f87916c1740 10 client.25891 fetch_fsmap learned FSMap version 33 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555d3fe0f9f0 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.778+0000 7f8771600640 1 -- 172.21.15.39:0/2963274554 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f876c0acb30 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f8771600640 1 -- 172.21.15.39:0/2963274554 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7f876c0b2c90 con 0x555d3f9beb70 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f87916c1740 10 client.25891 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f87916c1740 10 client.25891 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f87916c1740 20 client.25891 populate_metadata read hostname 'smithi039' 2024-09-21T01:11:56.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f87916c1740 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f973580 0x555d3fe12020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:11:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f87916c1740 4 client.25891 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}] 2024-09-21T01:11:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}) -- 0x555d3fcfa490 con 0x555d3f973580 2024-09-21T01:11:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.779+0000 7f8783e00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f973580 0x555d3fe12020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:11:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.780+0000 7f8771600640 10 client.25891 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.780+0000 7f8783e00640 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f973580 0x555d3fe12020 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=790a209daba00c13 server_cookie=acb0d10466147721 in_seq=0 out_seq=0 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f8771600640 1 -- 172.21.15.39:0/2963274554 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555d3fcfa490 con 0x555d3f973580 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f8771600640 10 client.25891 handle_command_reply: tid=0 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 client.25891 shutdown 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 msgr2=0x555d3fd97390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd97390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f876c0aa290 tx=0x7f876c0b1b70 comp rx=0 tx=0).stop 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 shutdown_connections 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x555d3f973580 0x555d3fe12020 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3f9df850 0x555d3fe0e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x555d3f9beb70 0x555d3fd97390 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 --2- 172.21.15.39:0/2963274554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d3f9b6990 0x555d3fd96e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 >> 172.21.15.39:0/2963274554 conn(0x555d3f9d3c40 msgr2=0x555d3fd87bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.849+0000 7f8771600640 0 client.25891 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.850+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 shutdown_connections 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.850+0000 7f87916c1740 1 -- 172.21.15.39:0/2963274554 wait complete. 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.850+0000 7f87916c1740 20 client.25891 trim_cache size 0 max 16384 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 33, 2024-09-21T01:11:56.851 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 34, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "b16db79e": { 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 34, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 141.1, 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:11:56.852 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:11:56.853 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f87680b66f0 msgr2=0x7f87680b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f87680b66f0 0x7f87680b8c00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f877c0303f0 tx=0x7f877c02c040 comp rx=0 tx=0).stop 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 msgr2=0x7f87841a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 0x7f87841a0e00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f876c00d950 tx=0x7f876c00de10 comp rx=0 tx=0).stop 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 shutdown_connections 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f87680b66f0 0x7f87680b8c00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878410be80 0x7f87841a60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f878410b530 0x7f87841a0e00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 --2- 172.21.15.39:0/4277383793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8784105bf0 0x7f87841a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 >> 172.21.15.39:0/4277383793 conn(0x7f8784101680 msgr2=0x7f8784108a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 shutdown_connections 2024-09-21T01:11:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:11:56.851+0000 7f8789400640 1 -- 172.21.15.39:0/4277383793 wait complete. 2024-09-21T01:11:57.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.56 MB/sec execute 264 sec latency 325.564 ms 2024-09-21T01:11:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:57 smithi039 ceph-mon[30415]: pgmap v646: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 571 KiB/s wr, 33 op/s 2024-09-21T01:11:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:57 smithi150 ceph-mon[34945]: pgmap v646: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 571 KiB/s wr, 33 op/s 2024-09-21T01:11:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:57 smithi160 ceph-mon[34696]: pgmap v646: 129 pgs: 129 active+clean; 232 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 571 KiB/s wr, 33 op/s 2024-09-21T01:11:58.056 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b16db79e', quiesce took 4.7 seconds. Will release after: 16.35 2024-09-21T01:11:58.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.54 MB/sec execute 265 sec latency 1325.624 ms 2024-09-21T01:11:59.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.53 MB/sec execute 266 sec latency 2325.752 ms 2024-09-21T01:11:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:11:59 smithi150 ceph-mon[34945]: pgmap v647: 129 pgs: 129 active+clean; 236 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-09-21T01:11:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:11:59 smithi160 ceph-mon[34696]: pgmap v647: 129 pgs: 129 active+clean; 236 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-09-21T01:11:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:11:59 smithi039 ceph-mon[30415]: pgmap v647: 129 pgs: 129 active+clean; 236 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-09-21T01:12:00.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.51 MB/sec execute 267 sec latency 3325.821 ms 2024-09-21T01:12:01.170 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.49 MB/sec execute 268 sec latency 4325.878 ms 2024-09-21T01:12:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:01 smithi150 ceph-mon[34945]: pgmap v648: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-09-21T01:12:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:12:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:01 smithi160 ceph-mon[34696]: pgmap v648: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-09-21T01:12:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:01 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:12:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:01 smithi039 ceph-mon[30415]: pgmap v648: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-09-21T01:12:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:12:02.174 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.48 MB/sec execute 269 sec latency 5330.252 ms 2024-09-21T01:12:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:12:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:12:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:12:02.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:12:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:12:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:12:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:12:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:12:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:12:03.174 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.46 MB/sec execute 270 sec latency 6330.354 ms 2024-09-21T01:12:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:03 smithi150 ceph-mon[34945]: pgmap v649: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-09-21T01:12:04.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:03 smithi160 ceph-mon[34696]: pgmap v649: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-09-21T01:12:04.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:03 smithi039 ceph-mon[30415]: pgmap v649: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-09-21T01:12:04.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.44 MB/sec execute 271 sec latency 7330.429 ms 2024-09-21T01:12:05.175 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.43 MB/sec execute 272 sec latency 8330.607 ms 2024-09-21T01:12:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:05 smithi150 ceph-mon[34945]: pgmap v650: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 102 op/s 2024-09-21T01:12:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:05 smithi150 ceph-mon[34945]: osdmap e95: 12 total, 12 up, 12 in 2024-09-21T01:12:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:05 smithi160 ceph-mon[34696]: pgmap v650: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 102 op/s 2024-09-21T01:12:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:05 smithi160 ceph-mon[34696]: osdmap e95: 12 total, 12 up, 12 in 2024-09-21T01:12:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:05 smithi039 ceph-mon[30415]: pgmap v650: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 102 op/s 2024-09-21T01:12:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:05 smithi039 ceph-mon[30415]: osdmap e95: 12 total, 12 up, 12 in 2024-09-21T01:12:06.175 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.41 MB/sec execute 273 sec latency 9330.700 ms 2024-09-21T01:12:07.175 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.39 MB/sec execute 274 sec latency 10330.842 ms 2024-09-21T01:12:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:07 smithi150 ceph-mon[34945]: pgmap v652: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-09-21T01:12:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:07 smithi160 ceph-mon[34696]: pgmap v652: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-09-21T01:12:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:07 smithi039 ceph-mon[30415]: pgmap v652: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-09-21T01:12:08.175 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.38 MB/sec execute 275 sec latency 11331.008 ms 2024-09-21T01:12:09.175 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.36 MB/sec execute 276 sec latency 12331.128 ms 2024-09-21T01:12:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:09 smithi150 ceph-mon[34945]: pgmap v653: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 608 KiB/s wr, 53 op/s 2024-09-21T01:12:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:09 smithi160 ceph-mon[34696]: pgmap v653: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 608 KiB/s wr, 53 op/s 2024-09-21T01:12:10.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:09 smithi039 ceph-mon[30415]: pgmap v653: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 608 KiB/s wr, 53 op/s 2024-09-21T01:12:10.175 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.35 MB/sec execute 277 sec latency 13331.249 ms 2024-09-21T01:12:11.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.33 MB/sec execute 278 sec latency 14331.390 ms 2024-09-21T01:12:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:11 smithi150 ceph-mon[34945]: pgmap v654: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 443 KiB/s wr, 48 op/s 2024-09-21T01:12:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:11 smithi160 ceph-mon[34696]: pgmap v654: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 443 KiB/s wr, 48 op/s 2024-09-21T01:12:12.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:11 smithi039 ceph-mon[30415]: pgmap v654: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 443 KiB/s wr, 48 op/s 2024-09-21T01:12:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.32 MB/sec execute 279 sec latency 15331.531 ms 2024-09-21T01:12:13.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.30 MB/sec execute 280 sec latency 16331.714 ms 2024-09-21T01:12:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:13 smithi150 ceph-mon[34945]: pgmap v655: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 382 KiB/s wr, 41 op/s 2024-09-21T01:12:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:13 smithi160 ceph-mon[34696]: pgmap v655: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 382 KiB/s wr, 41 op/s 2024-09-21T01:12:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:13 smithi039 ceph-mon[30415]: pgmap v655: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 382 KiB/s wr, 41 op/s 2024-09-21T01:12:14.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.29 MB/sec execute 281 sec latency 17331.880 ms 2024-09-21T01:12:14.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:12:14.806 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:12:15.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.27 MB/sec execute 282 sec latency 18331.942 ms 2024-09-21T01:12:15.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:15 smithi039 ceph-mon[30415]: pgmap v656: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 211 KiB/s wr, 18 op/s 2024-09-21T01:12:15.480 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:15 smithi039 ceph-mon[30415]: osdmap e96: 12 total, 12 up, 12 in 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 -- 172.21.15.39:0/3834208701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a641092f0 msgr2=0x7f6a64109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/3834208701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a641092f0 0x7f6a64109700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009900 tx=0x7f6a5002a410 comp rx=0 tx=0).stop 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 -- 172.21.15.39:0/3834208701 shutdown_connections 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/3834208701 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a64109c40 0x7f6a64110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/3834208701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a641092f0 0x7f6a64109700 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/3834208701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a641039b0 0x7f6a64103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 -- 172.21.15.39:0/3834208701 >> 172.21.15.39:0/3834208701 conn(0x7f6a640ff440 msgr2=0x7f6a64101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:15.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.544+0000 7f6a6aa00640 1 -- 172.21.15.39:0/3834208701 shutdown_connections 2024-09-21T01:12:15.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.545+0000 7f6a6aa00640 1 -- 172.21.15.39:0/3834208701 wait complete. 2024-09-21T01:12:15.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 Processor -- start 2024-09-21T01:12:15.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 -- start start 2024-09-21T01:12:15.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a641039b0 0x7f6a64075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:15.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a641092f0 0x7f6a64075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:15.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a64109c40 0x7f6a64076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:15.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64113950 con 0x7f6a64109c40 2024-09-21T01:12:15.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a641137d0 con 0x7f6a641092f0 2024-09-21T01:12:15.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6aa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64113ad0 con 0x7f6a641039b0 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a6a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a64109c40 0x7f6a64076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a69600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a641039b0 0x7f6a64075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.546+0000 7f6a69600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a641039b0 0x7f6a64075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:50210/0 (socket says 172.21.15.39:50210) 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a69600640 1 -- 172.21.15.39:0/2943972835 learned_addr learned my addr 172.21.15.39:0/2943972835 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a68c00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a641092f0 0x7f6a64075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a6a000640 1 -- 172.21.15.39:0/2943972835 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a641039b0 msgr2=0x7f6a64075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a6a000640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a641039b0 0x7f6a64075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a6a000640 1 -- 172.21.15.39:0/2943972835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a641092f0 msgr2=0x7f6a64075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a6a000640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a641092f0 0x7f6a64075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a6a000640 1 -- 172.21.15.39:0/2943972835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641a85a0 con 0x7f6a64109c40 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.547+0000 7f6a6a000640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a64109c40 0x7f6a64076190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009fd0 tx=0x7f6a5c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.548+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00ce20 con 0x7f6a64109c40 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.548+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641a8890 con 0x7f6a64109c40 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.548+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a5c005020 con 0x7f6a64109c40 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.548+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c010690 con 0x7f6a64109c40 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.548+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641a8df0 con 0x7f6a64109c40 2024-09-21T01:12:15.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.549+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a64103dc0 con 0x7f6a64109c40 2024-09-21T01:12:15.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.552+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6a5c0054d0 con 0x7f6a64109c40 2024-09-21T01:12:15.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.554+0000 7f6a62000640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6a4c0b6680 0x7f6a4c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:15.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.555+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9511+0+0 (secure 0 0 0) 0x7f6a5c0e8210 con 0x7f6a64109c40 2024-09-21T01:12:15.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.555+0000 7f6a69600640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6a4c0b6680 0x7f6a4c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:15.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.555+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6a5c0ae320 con 0x7f6a64109c40 2024-09-21T01:12:15.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.556+0000 7f6a69600640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6a4c0b6680 0x7f6a4c0b8b90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6a64077160 tx=0x7f6a58005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.791+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a6402e640 con 0x7f6a64109c40 2024-09-21T01:12:15.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.792+0000 7f6a62000640 1 -- 172.21.15.39:0/2943972835 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11915 (secure 0 0 0) 0x7f6a5c0b2700 con 0x7f6a64109c40 2024-09-21T01:12:15.794 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:12:15.794 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":33,"btime":"2024-09-21T01:11:30:411273+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:11:29.517648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:12:15.794 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 33 2024-09-21T01:12:15.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.794+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6a4c0b6680 msgr2=0x7f6a4c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:15.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.794+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6a4c0b6680 0x7f6a4c0b8b90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6a64077160 tx=0x7f6a58005f90 comp rx=0 tx=0).stop 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.794+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a64109c40 msgr2=0x7f6a64076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.794+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a64109c40 0x7f6a64076190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009fd0 tx=0x7f6a5c00ef10 comp rx=0 tx=0).stop 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 shutdown_connections 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a64109c40 0x7f6a64076190 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6a4c0b6680 0x7f6a4c0b8b90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a641092f0 0x7f6a64075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 --2- 172.21.15.39:0/2943972835 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6a641039b0 0x7f6a64075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 >> 172.21.15.39:0/2943972835 conn(0x7f6a640ff440 msgr2=0x7f6a64101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 shutdown_connections 2024-09-21T01:12:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:15.795+0000 7f6a6aa00640 1 -- 172.21.15.39:0/2943972835 wait complete. 2024-09-21T01:12:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:15 smithi150 ceph-mon[34945]: pgmap v656: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 211 KiB/s wr, 18 op/s 2024-09-21T01:12:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:15 smithi150 ceph-mon[34945]: osdmap e96: 12 total, 12 up, 12 in 2024-09-21T01:12:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:15 smithi160 ceph-mon[34696]: pgmap v656: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 211 KiB/s wr, 18 op/s 2024-09-21T01:12:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:15 smithi160 ceph-mon[34696]: osdmap e96: 12 total, 12 up, 12 in 2024-09-21T01:12:16.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.25 MB/sec execute 283 sec latency 19332.066 ms 2024-09-21T01:12:16.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:16 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2943972835' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:12:16.763 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id b16db79e --release --await' 2024-09-21T01:12:16.763 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id b16db79e --release --await 2024-09-21T01:12:16.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:16 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2943972835' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:12:16.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:16 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2943972835' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:12:17.142 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:12:17.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.24 MB/sec execute 284 sec latency 20332.124 ms 2024-09-21T01:12:17.533 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:17 smithi039 ceph-mon[30415]: pgmap v658: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 198 KiB/s wr, 16 op/s 2024-09-21T01:12:17.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:17 smithi150 ceph-mon[34945]: pgmap v658: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 198 KiB/s wr, 16 op/s 2024-09-21T01:12:17.757 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:17 smithi160 ceph-mon[34696]: pgmap v658: 129 pgs: 129 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 198 KiB/s wr, 16 op/s 2024-09-21T01:12:18.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 363486 4.23 MB/sec execute 285 sec latency 21332.168 ms 2024-09-21T01:12:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 -- 172.21.15.39:0/299317241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b0103b60 msgr2=0x7ff0b0103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 --2- 172.21.15.39:0/299317241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b0103b60 0x7ff0b0103f70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009ca0 tx=0x7ff0a002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 -- 172.21.15.39:0/299317241 shutdown_connections 2024-09-21T01:12:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 --2- 172.21.15.39:0/299317241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0109df0 0x7ff0b0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 --2- 172.21.15.39:0/299317241 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0b01094a0 0x7ff0b01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 --2- 172.21.15.39:0/299317241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b0103b60 0x7ff0b0103f70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 -- 172.21.15.39:0/299317241 >> 172.21.15.39:0/299317241 conn(0x7ff0b007b850 msgr2=0x7ff0b007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 -- 172.21.15.39:0/299317241 shutdown_connections 2024-09-21T01:12:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.352+0000 7ff0b6000640 1 -- 172.21.15.39:0/299317241 wait complete. 2024-09-21T01:12:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.353+0000 7ff0b6000640 1 Processor -- start 2024-09-21T01:12:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.353+0000 7ff0b6000640 1 -- start start 2024-09-21T01:12:18.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.353+0000 7ff0b6000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0103b60 0x7ff0b019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.353+0000 7ff0b6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 0x7ff0b019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0b6000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0b0109df0 0x7ff0b019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0b6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b0113ab0 con 0x7ff0b01094a0 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0b6000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b0113930 con 0x7ff0b0103b60 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0b6000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b0113c30 con 0x7ff0b0109df0 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 0x7ff0b019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0af400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0103b60 0x7ff0b019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 0x7ff0b019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53432/0 (socket says 172.21.15.39:53432) 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0af400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0103b60 0x7ff0b019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50636/0 (socket says 172.21.15.39:50636) 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 -- 172.21.15.39:0/1649869142 learned_addr learned my addr 172.21.15.39:0/1649869142 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0afe00640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0b0109df0 0x7ff0b019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 -- 172.21.15.39:0/1649869142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0b0109df0 msgr2=0x7ff0b019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0b0109df0 0x7ff0b019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 -- 172.21.15.39:0/1649869142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0103b60 msgr2=0x7ff0b019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0103b60 0x7ff0b019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.354+0000 7ff0a6a00640 1 -- 172.21.15.39:0/1649869142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b01a0430 con 0x7ff0b01094a0 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.355+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 0x7ff0b019aa60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff09800e9c0 tx=0x7ff09800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.355+0000 7ff0acc00640 1 -- 172.21.15.39:0/1649869142 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09800ccc0 con 0x7ff0b01094a0 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.355+0000 7ff0acc00640 1 -- 172.21.15.39:0/1649869142 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0980054d0 con 0x7ff0b01094a0 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.355+0000 7ff0acc00640 1 -- 172.21.15.39:0/1649869142 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098010430 con 0x7ff0b01094a0 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.355+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b01a0720 con 0x7ff0b01094a0 2024-09-21T01:12:18.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.355+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b01a6e80 con 0x7ff0b01094a0 2024-09-21T01:12:18.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.358+0000 7ff0bcbd9740 1 Processor -- start 2024-09-21T01:12:18.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 -- start start 2024-09-21T01:12:18.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0acc00640 1 -- 172.21.15.39:0/1649869142 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff098004f40 con 0x7ff0b01094a0 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 0x55eb1b36a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b36a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afa2840 0x55eb1b36ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55eb1b2e1290 con 0x55eb1afa2840 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55eb1b2d06c0 con 0x55eb1af9a660 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eb1b2d44d0 con 0x55eb1afc3520 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 0x55eb1b36a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0a6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afa2840 0x55eb1b36ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 0x55eb1b36a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:50234/0 (socket says 172.21.15.39:50234) 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0afe00640 1 -- 172.21.15.39:0/2156829668 learned_addr learned my addr 172.21.15.39:0/2156829668 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.359+0000 7ff0af400640 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b36a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0afe00640 1 -- 172.21.15.39:0/2156829668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 msgr2=0x55eb1b36a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0afe00640 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b36a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0afe00640 1 -- 172.21.15.39:0/2156829668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afa2840 msgr2=0x55eb1b36ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0afe00640 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afa2840 0x55eb1b36ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0afe00640 1 -- 172.21.15.39:0/2156829668 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb1b25d460 con 0x55eb1afc3520 2024-09-21T01:12:18.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0af400640 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b36a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:12:18.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.360+0000 7ff0afe00640 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 0x55eb1b36a1c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009c90 tx=0x7ff0a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b937cff6198fca49 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.361+0000 7ff09fe00640 1 -- 172.21.15.39:0/2156829668 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8004230 con 0x55eb1afc3520 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.361+0000 7ff09fe00640 1 -- 172.21.15.39:0/2156829668 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0a8009e40 con 0x55eb1afc3520 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.361+0000 7ff09fe00640 1 -- 172.21.15.39:0/2156829668 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8019710 con 0x55eb1afc3520 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.361+0000 7ff0acc00640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0940b6680 0x7ff0940b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.361+0000 7ff0acc00640 1 -- 172.21.15.39:0/1649869142 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9511+0+0 (secure 0 0 0) 0x7ff0980e7fe0 con 0x7ff0b01094a0 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.361+0000 7ff0af400640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0940b6680 0x7ff0940b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/2156829668 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 msgr2=0x55eb1b36a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 0x55eb1b36a1c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009c90 tx=0x7ff0a8011810 comp rx=0 tx=0).stop 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/2156829668 shutdown_connections 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afa2840 0x55eb1b36ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b36a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/2156829668 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afc3520 0x55eb1b36a1c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/2156829668 >> 172.21.15.39:0/2156829668 conn(0x55eb1afb7910 msgr2=0x55eb1b3680b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:18.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/2156829668 shutdown_connections 2024-09-21T01:12:18.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/2156829668 wait complete. 2024-09-21T01:12:18.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.362+0000 7ff0af400640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0940b6680 0x7ff0940b8b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55eb1b368e40 tx=0x7ff0a0009360 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:18.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.363+0000 7ff0bcbd9740 1 -- start start 2024-09-21T01:12:18.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.363+0000 7ff0bcbd9740 1 Processor -- start 2024-09-21T01:12:18.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 0x55eb1b375cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afa2840 0x55eb1b376230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b381cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55eb1b131bb0 con 0x55eb1afc3520 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55eb1b26b6c0 con 0x55eb1afa2840 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eb1b3789a0 con 0x55eb1af9a660 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0bcbd9740 10 client.0 _ll_register_callbacks cb 0x55eb1b286490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0af400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 0x55eb1b375cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0af400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 0x55eb1b375cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:50246/0 (socket says 172.21.15.39:50246) 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0af400640 1 -- 172.21.15.39:0/1553557640 learned_addr learned my addr 172.21.15.39:0/1553557640 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0afe00640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b381cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afa2840 0x55eb1b376230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0af400640 1 -- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afa2840 msgr2=0x55eb1b376230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.364+0000 7ff0af400640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afa2840 0x55eb1b376230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff0af400640 1 -- 172.21.15.39:0/1553557640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 msgr2=0x55eb1b381cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff0af400640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b381cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff0af400640 1 -- 172.21.15.39:0/1553557640 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb1b02eb00 con 0x55eb1af9a660 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff0af400640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 0x55eb1b375cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0035000 tx=0x7ff0a002a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6a05b40b91020d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff09cc00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff09cc00640 1 -- 172.21.15.39:0/1553557640 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0003d00 con 0x55eb1af9a660 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff09cc00640 1 -- 172.21.15.39:0/1553557640 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9511+0+0 (secure 0 0 0) 0x7ff0a0040820 con 0x55eb1af9a660 2024-09-21T01:12:18.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.365+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eb1b2e31e0 con 0x55eb1af9a660 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff09cc00640 1 -- 172.21.15.39:0/1553557640 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff0a0040a20 con 0x55eb1af9a660 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 10 client.25909 fetch_fsmap learned FSMap version 33 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55eb1b0a5960 con 0x55eb1af9a660 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff09cc00640 1 -- 172.21.15.39:0/1553557640 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55eb1b2e31e0 con 0x55eb1af9a660 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff09cc00640 1 -- 172.21.15.39:0/1553557640 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7ff0a0055070 con 0x55eb1af9a660 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 10 client.25909 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 10 client.25909 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 20 client.25909 populate_metadata read hostname 'smithi039' 2024-09-21T01:12:18.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af81ea0 0x55eb1b387c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 4 client.25909 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:12:18.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.366+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb1b2dec90 con 0x55eb1af81ea0 2024-09-21T01:12:18.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.367+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af81ea0 0x55eb1b387c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.367+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af81ea0 0x55eb1b387c60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d717a43445dfc1cc server_cookie=b7d8bc1d0cec0127 in_seq=0 out_seq=0 2024-09-21T01:12:18.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.367+0000 7ff09cc00640 10 client.25909 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.368+0000 7ff09cc00640 1 -- 172.21.15.39:0/1553557640 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55eb1b2dec90 con 0x55eb1af81ea0 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.368+0000 7ff09cc00640 10 client.25909 handle_command_reply: tid=0 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.368+0000 7ff0bcbd9740 1 client.25909 shutdown 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 msgr2=0x55eb1b375cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 0x55eb1b375cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0035000 tx=0x7ff0a002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 shutdown_connections 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af81ea0 0x55eb1b387c60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b381cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afa2840 0x55eb1b376230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1553557640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1af9a660 0x55eb1b375cf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 >> 172.21.15.39:0/1553557640 conn(0x55eb1afb7910 msgr2=0x55eb1b36d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:18.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff09cc00640 0 client.25909 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:12:18.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 shutdown_connections 2024-09-21T01:12:18.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1553557640 wait complete. 2024-09-21T01:12:18.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.369+0000 7ff0bcbd9740 20 client.25909 trim_cache size 0 max 16384 2024-09-21T01:12:18.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.387+0000 7ff0bcbd9740 1 Processor -- start 2024-09-21T01:12:18.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.387+0000 7ff0bcbd9740 1 -- start start 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b3702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b3779d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b36b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b01b16a0 con 0x55eb1afc3520 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b01aef10 con 0x55eb1af9a660 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b01acd80 con 0x55eb1afa2840 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b36b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b36b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53460/0 (socket says 172.21.15.39:53460) 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0afe00640 1 -- 172.21.15.39:0/1164539434 learned_addr learned my addr 172.21.15.39:0/1164539434 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0af400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b3702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b3779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0af400640 1 -- 172.21.15.39:0/1164539434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 msgr2=0x55eb1b3779d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0af400640 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b3779d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0af400640 1 -- 172.21.15.39:0/1164539434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 msgr2=0x55eb1b36b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0af400640 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b36b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.388+0000 7ff0af400640 1 -- 172.21.15.39:0/1164539434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb1b368e70 con 0x55eb1af9a660 2024-09-21T01:12:18.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.389+0000 7ff0af400640 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b3702e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0004850 tx=0x7ff0a0009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c7e6faaafb189c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.389+0000 7ff09f400640 1 -- 172.21.15.39:0/1164539434 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0040820 con 0x55eb1af9a660 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.389+0000 7ff09f400640 1 -- 172.21.15.39:0/1164539434 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0a0036050 con 0x55eb1af9a660 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.389+0000 7ff09f400640 1 -- 172.21.15.39:0/1164539434 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0030820 con 0x55eb1af9a660 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1164539434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 msgr2=0x55eb1b3702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b3702e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0004850 tx=0x7ff0a0009c40 comp rx=0 tx=0).stop 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1164539434 shutdown_connections 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1afc3520 0x55eb1b36b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b3779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/1164539434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1af9a660 0x55eb1b3702e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1164539434 >> 172.21.15.39:0/1164539434 conn(0x55eb1afb7910 msgr2=0x55eb1b36b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1164539434 shutdown_connections 2024-09-21T01:12:18.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/1164539434 wait complete. 2024-09-21T01:12:18.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.390+0000 7ff0bcbd9740 1 -- start start 2024-09-21T01:12:18.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 Processor -- start 2024-09-21T01:12:18.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 0x55eb1b37a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b37ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afc3520 0x55eb1b3eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b01ad3d0 con 0x55eb1af9a660 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b01ae840 con 0x55eb1afc3520 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b01adaa0 con 0x55eb1afa2840 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.391+0000 7ff0bcbd9740 10 client.0 _ll_register_callbacks cb 0x7ff0a0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 0x55eb1b37a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afc3520 0x55eb1b3eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afc3520 0x55eb1b3eb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50678/0 (socket says 172.21.15.39:50678) 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 0x55eb1b37a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53474/0 (socket says 172.21.15.39:53474) 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0afe00640 1 -- 172.21.15.39:0/3656648619 learned_addr learned my addr 172.21.15.39:0/3656648619 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b37ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 -- 172.21.15.39:0/3656648619 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 msgr2=0x55eb1b37ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b37ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 -- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afc3520 msgr2=0x55eb1b3eb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afc3520 0x55eb1b3eb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 -- 172.21.15.39:0/3656648619 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb1b3ebd80 con 0x55eb1af9a660 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff0af400640 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 0x55eb1b37a730 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0005e30 tx=0x7ff0a0040df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4d0a9844b3346b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.392+0000 7ff09cc00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.393+0000 7ff09cc00640 1 -- 172.21.15.39:0/3656648619 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a00318e0 con 0x55eb1af9a660 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.393+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0a0036050 con 0x55eb1af9a660 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.393+0000 7ff09cc00640 1 -- 172.21.15.39:0/3656648619 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9511+0+0 (secure 0 0 0) 0x7ff0a0046030 con 0x55eb1af9a660 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.393+0000 7ff09cc00640 1 -- 172.21.15.39:0/3656648619 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff0a0040820 con 0x55eb1af9a660 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.393+0000 7ff0bcbd9740 10 client.15897 fetch_fsmap learned FSMap version 33 2024-09-21T01:12:18.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.393+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55eb1b3f31b0 con 0x55eb1af9a660 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff09cc00640 1 -- 172.21.15.39:0/3656648619 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0a0036050 con 0x55eb1af9a660 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff09cc00640 1 -- 172.21.15.39:0/3656648619 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) ==== 8366+0+0 (secure 0 0 0) 0x7ff0a0055070 con 0x55eb1af9a660 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0bcbd9740 10 client.15897 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0bcbd9740 10 client.15897 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0bcbd9740 20 client.15897 populate_metadata read hostname 'smithi039' 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af57250 0x55eb1b3f5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0bcbd9740 4 client.15897 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b16db79e", "release": true, "await": true}] 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b16db79e", "release": true, "await": true}) -- 0x55eb1b2dc0c0 con 0x55eb1af57250 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.394+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af57250 0x55eb1b3f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.395+0000 7ff0a6a00640 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af57250 0x55eb1b3f5700 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec2f4976784fdf8 server_cookie=5b9f012fd9934c74 in_seq=0 out_seq=0 2024-09-21T01:12:18.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.395+0000 7ff09cc00640 10 client.15897 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:12:18.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff09cc00640 1 -- 172.21.15.39:0/3656648619 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55eb1b2dc0c0 con 0x55eb1af57250 2024-09-21T01:12:18.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff09cc00640 10 client.15897 handle_command_reply: tid=0 2024-09-21T01:12:18.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 client.15897 shutdown 2024-09-21T01:12:18.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 msgr2=0x55eb1b37a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 0x55eb1b37a730 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0005e30 tx=0x7ff0a0040df0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 shutdown_connections 2024-09-21T01:12:18.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eb1af57250 0x55eb1b3f5700 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1afc3520 0x55eb1b3eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eb1afa2840 0x55eb1b37ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 --2- 172.21.15.39:0/3656648619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb1af9a660 0x55eb1b37a730 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff09cc00640 0 client.15897 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 >> 172.21.15.39:0/3656648619 conn(0x55eb1afb7910 msgr2=0x55eb1b377730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.405+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 shutdown_connections 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.406+0000 7ff0bcbd9740 1 -- 172.21.15.39:0/3656648619 wait complete. 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.406+0000 7ff0bcbd9740 20 client.15897 trim_cache size 0 max 16384 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 33, 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 36, 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:12:18.408 INFO:teuthology.orchestra.run.smithi039.stdout: "b16db79e": { 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 36, 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 141.1, 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:12:18.409 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.406+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0940b6680 msgr2=0x7ff0940b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.406+0000 7ff0b6000640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0940b6680 0x7ff0940b8b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55eb1b368e40 tx=0x7ff0a0009360 comp rx=0 tx=0).stop 2024-09-21T01:12:18.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.406+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 msgr2=0x7ff0b019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.406+0000 7ff0b6000640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 0x7ff0b019aa60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff09800e9c0 tx=0x7ff09800edc0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 shutdown_connections 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0b0109df0 0x7ff0b019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0b01094a0 0x7ff0b019aa60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0940b6680 0x7ff0940b8b90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 --2- 172.21.15.39:0/1649869142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0103b60 0x7ff0b019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 >> 172.21.15.39:0/1649869142 conn(0x7ff0b007b850 msgr2=0x7ff0b0107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 shutdown_connections 2024-09-21T01:12:18.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:12:18.407+0000 7ff0b6000640 1 -- 172.21.15.39:0/1649869142 wait complete. 2024-09-21T01:12:19.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365334 4.25 MB/sec execute 286 sec latency 21564.362 ms 2024-09-21T01:12:19.527 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b16db79e', total seconds elapsed: 26.2 2024-09-21T01:12:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:19 smithi150 ceph-mon[34945]: pgmap v659: 129 pgs: 129 active+clean; 235 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 109 KiB/s wr, 11 op/s 2024-09-21T01:12:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:19 smithi160 ceph-mon[34696]: pgmap v659: 129 pgs: 129 active+clean; 235 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 109 KiB/s wr, 11 op/s 2024-09-21T01:12:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:19 smithi039 ceph-mon[30415]: pgmap v659: 129 pgs: 129 active+clean; 235 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 109 KiB/s wr, 11 op/s 2024-09-21T01:12:20.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.24 MB/sec execute 287 sec latency 466.556 ms 2024-09-21T01:12:21.176 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.23 MB/sec execute 288 sec latency 1466.690 ms 2024-09-21T01:12:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:21 smithi150 ceph-mon[34945]: pgmap v660: 129 pgs: 129 active+clean; 235 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 309 KiB/s wr, 15 op/s 2024-09-21T01:12:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:21 smithi160 ceph-mon[34696]: pgmap v660: 129 pgs: 129 active+clean; 235 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 309 KiB/s wr, 15 op/s 2024-09-21T01:12:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:21 smithi039 ceph-mon[30415]: pgmap v660: 129 pgs: 129 active+clean; 235 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 309 KiB/s wr, 15 op/s 2024-09-21T01:12:22.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.21 MB/sec execute 289 sec latency 2466.835 ms 2024-09-21T01:12:23.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.20 MB/sec execute 290 sec latency 3466.986 ms 2024-09-21T01:12:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:23 smithi150 ceph-mon[34945]: pgmap v661: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 650 KiB/s wr, 36 op/s 2024-09-21T01:12:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:23 smithi160 ceph-mon[34696]: pgmap v661: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 650 KiB/s wr, 36 op/s 2024-09-21T01:12:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:23 smithi039 ceph-mon[30415]: pgmap v661: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 650 KiB/s wr, 36 op/s 2024-09-21T01:12:24.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.19 MB/sec execute 291 sec latency 4467.065 ms 2024-09-21T01:12:25.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.17 MB/sec execute 292 sec latency 5467.197 ms 2024-09-21T01:12:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:25 smithi150 ceph-mon[34945]: pgmap v662: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 718 KiB/s wr, 48 op/s 2024-09-21T01:12:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:25 smithi160 ceph-mon[34696]: pgmap v662: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 718 KiB/s wr, 48 op/s 2024-09-21T01:12:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:25 smithi039 ceph-mon[30415]: pgmap v662: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 718 KiB/s wr, 48 op/s 2024-09-21T01:12:26.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.16 MB/sec execute 293 sec latency 6467.327 ms 2024-09-21T01:12:26.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:26 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:12:26.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:26 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:12:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:26 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:12:27.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.14 MB/sec execute 294 sec latency 7467.478 ms 2024-09-21T01:12:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:27 smithi150 ceph-mon[34945]: pgmap v663: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 612 KiB/s wr, 43 op/s 2024-09-21T01:12:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:27 smithi160 ceph-mon[34696]: pgmap v663: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 612 KiB/s wr, 43 op/s 2024-09-21T01:12:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:27 smithi039 ceph-mon[30415]: pgmap v663: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 612 KiB/s wr, 43 op/s 2024-09-21T01:12:28.177 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.13 MB/sec execute 295 sec latency 8467.609 ms 2024-09-21T01:12:29.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 365942 4.11 MB/sec execute 296 sec latency 9467.814 ms 2024-09-21T01:12:29.705 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:29 smithi039 ceph-mon[30415]: pgmap v664: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 626 KiB/s wr, 46 op/s 2024-09-21T01:12:29.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:29 smithi150 ceph-mon[34945]: pgmap v664: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 626 KiB/s wr, 46 op/s 2024-09-21T01:12:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:29 smithi160 ceph-mon[34696]: pgmap v664: 129 pgs: 129 active+clean; 236 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 626 KiB/s wr, 46 op/s 2024-09-21T01:12:30.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 366846 4.12 MB/sec execute 297 sec latency 9855.376 ms 2024-09-21T01:12:31.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 367032 4.10 MB/sec execute 298 sec latency 854.818 ms 2024-09-21T01:12:31.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:31 smithi039 ceph-mon[30415]: pgmap v665: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 547 KiB/s wr, 43 op/s 2024-09-21T01:12:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:31 smithi150 ceph-mon[34945]: pgmap v665: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 547 KiB/s wr, 43 op/s 2024-09-21T01:12:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:31 smithi160 ceph-mon[34696]: pgmap v665: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 547 KiB/s wr, 43 op/s 2024-09-21T01:12:32.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 367524 4.09 MB/sec execute 299 sec latency 1668.756 ms 2024-09-21T01:12:33.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 369409 4.12 MB/sec execute 300 sec latency 11.778 ms 2024-09-21T01:12:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:33 smithi150 ceph-mon[34945]: pgmap v666: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 462 KiB/s wr, 49 op/s 2024-09-21T01:12:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:33 smithi160 ceph-mon[34696]: pgmap v666: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 462 KiB/s wr, 49 op/s 2024-09-21T01:12:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:33 smithi039 ceph-mon[30415]: pgmap v666: 129 pgs: 129 active+clean; 237 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 462 KiB/s wr, 49 op/s 2024-09-21T01:12:34.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 370551 4.13 MB/sec execute 301 sec latency 18.503 ms 2024-09-21T01:12:35.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 372546 4.16 MB/sec execute 302 sec latency 188.416 ms 2024-09-21T01:12:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:35 smithi150 ceph-mon[34945]: pgmap v667: 129 pgs: 129 active+clean; 238 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 241 KiB/s wr, 39 op/s 2024-09-21T01:12:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:35 smithi160 ceph-mon[34696]: pgmap v667: 129 pgs: 129 active+clean; 238 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 241 KiB/s wr, 39 op/s 2024-09-21T01:12:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:35 smithi039 ceph-mon[30415]: pgmap v667: 129 pgs: 129 active+clean; 238 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 241 KiB/s wr, 39 op/s 2024-09-21T01:12:36.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 373884 4.16 MB/sec execute 303 sec latency 22.076 ms 2024-09-21T01:12:37.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 374673 4.15 MB/sec execute 304 sec latency 636.767 ms 2024-09-21T01:12:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:37 smithi150 ceph-mon[34945]: pgmap v668: 129 pgs: 129 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 247 KiB/s wr, 39 op/s 2024-09-21T01:12:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:37 smithi160 ceph-mon[34696]: pgmap v668: 129 pgs: 129 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 247 KiB/s wr, 39 op/s 2024-09-21T01:12:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:37 smithi039 ceph-mon[30415]: pgmap v668: 129 pgs: 129 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 247 KiB/s wr, 39 op/s 2024-09-21T01:12:38.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 376566 4.18 MB/sec execute 305 sec latency 10.987 ms 2024-09-21T01:12:39.178 INFO:tasks.workunit.client.0.smithi039.stdout: 1 377666 4.19 MB/sec execute 306 sec latency 20.125 ms 2024-09-21T01:12:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:39 smithi150 ceph-mon[34945]: pgmap v669: 129 pgs: 129 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 825 KiB/s wr, 91 op/s 2024-09-21T01:12:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:39 smithi160 ceph-mon[34696]: pgmap v669: 129 pgs: 129 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 825 KiB/s wr, 91 op/s 2024-09-21T01:12:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:39 smithi039 ceph-mon[30415]: pgmap v669: 129 pgs: 129 active+clean; 245 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 825 KiB/s wr, 91 op/s 2024-09-21T01:12:40.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 377666 4.17 MB/sec execute 307 sec latency 1018.846 ms 2024-09-21T01:12:41.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 377666 4.16 MB/sec execute 308 sec latency 2019.006 ms 2024-09-21T01:12:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:41 smithi150 ceph-mon[34945]: pgmap v670: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-09-21T01:12:41.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:41 smithi160 ceph-mon[34696]: pgmap v670: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-09-21T01:12:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:41 smithi039 ceph-mon[30415]: pgmap v670: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-09-21T01:12:42.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 378195 4.15 MB/sec execute 309 sec latency 2831.581 ms 2024-09-21T01:12:43.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 380114 4.18 MB/sec execute 310 sec latency 12.545 ms 2024-09-21T01:12:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:43 smithi150 ceph-mon[34945]: pgmap v671: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-09-21T01:12:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:43 smithi160 ceph-mon[34696]: pgmap v671: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-09-21T01:12:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:43 smithi039 ceph-mon[30415]: pgmap v671: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-09-21T01:12:44.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 381231 4.18 MB/sec execute 311 sec latency 18.540 ms 2024-09-21T01:12:45.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 382848 4.20 MB/sec execute 312 sec latency 343.737 ms 2024-09-21T01:12:45.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:45 smithi150 ceph-mon[34945]: pgmap v672: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-09-21T01:12:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:45 smithi160 ceph-mon[34696]: pgmap v672: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-09-21T01:12:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:45 smithi039 ceph-mon[30415]: pgmap v672: 129 pgs: 129 active+clean; 248 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-09-21T01:12:46.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 384161 4.20 MB/sec execute 313 sec latency 18.272 ms 2024-09-21T01:12:47.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 385267 4.20 MB/sec execute 314 sec latency 405.817 ms 2024-09-21T01:12:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:47 smithi150 ceph-mon[34945]: pgmap v673: 129 pgs: 129 active+clean; 252 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-09-21T01:12:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:47 smithi160 ceph-mon[34696]: pgmap v673: 129 pgs: 129 active+clean; 252 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-09-21T01:12:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:47 smithi039 ceph-mon[30415]: pgmap v673: 129 pgs: 129 active+clean; 252 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-09-21T01:12:48.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 387075 4.23 MB/sec execute 315 sec latency 11.051 ms 2024-09-21T01:12:49.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 388235 4.23 MB/sec execute 316 sec latency 16.894 ms 2024-09-21T01:12:49.511 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:49 smithi039 ceph-mon[30415]: pgmap v674: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2024-09-21T01:12:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:49 smithi150 ceph-mon[34945]: pgmap v674: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2024-09-21T01:12:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:49 smithi160 ceph-mon[34696]: pgmap v674: 129 pgs: 129 active+clean; 255 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2024-09-21T01:12:50.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 388568 4.22 MB/sec execute 317 sec latency 495.240 ms 2024-09-21T01:12:51.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 390723 4.25 MB/sec execute 318 sec latency 12.220 ms 2024-09-21T01:12:51.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:51 smithi150 ceph-mon[34945]: pgmap v675: 129 pgs: 129 active+clean; 257 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-09-21T01:12:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:51 smithi160 ceph-mon[34696]: pgmap v675: 129 pgs: 129 active+clean; 257 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-09-21T01:12:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:51 smithi039 ceph-mon[30415]: pgmap v675: 129 pgs: 129 active+clean; 257 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-09-21T01:12:52.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 391911 4.25 MB/sec execute 319 sec latency 146.584 ms 2024-09-21T01:12:53.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 391911 4.24 MB/sec execute 320 sec latency 1146.720 ms 2024-09-21T01:12:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:53 smithi150 ceph-mon[34945]: pgmap v676: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-09-21T01:12:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:53 smithi160 ceph-mon[34696]: pgmap v676: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-09-21T01:12:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:53 smithi039 ceph-mon[30415]: pgmap v676: 129 pgs: 129 active+clean; 260 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-09-21T01:12:54.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 391911 4.23 MB/sec execute 321 sec latency 2146.808 ms 2024-09-21T01:12:55.179 INFO:tasks.workunit.client.0.smithi039.stdout: 1 393575 4.24 MB/sec execute 322 sec latency 2500.009 ms 2024-09-21T01:12:55.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:55 smithi150 ceph-mon[34945]: pgmap v677: 129 pgs: 129 active+clean; 262 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-09-21T01:12:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:55 smithi160 ceph-mon[34696]: pgmap v677: 129 pgs: 129 active+clean; 262 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-09-21T01:12:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:55 smithi039 ceph-mon[30415]: pgmap v677: 129 pgs: 129 active+clean; 262 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-09-21T01:12:56.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 394951 4.25 MB/sec execute 323 sec latency 21.309 ms 2024-09-21T01:12:57.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 395637 4.25 MB/sec execute 324 sec latency 334.379 ms 2024-09-21T01:12:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:57 smithi150 ceph-mon[34945]: pgmap v678: 129 pgs: 129 active+clean; 263 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T01:12:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:57 smithi160 ceph-mon[34696]: pgmap v678: 129 pgs: 129 active+clean; 263 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T01:12:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:57 smithi039 ceph-mon[30415]: pgmap v678: 129 pgs: 129 active+clean; 263 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T01:12:58.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 397685 4.28 MB/sec execute 325 sec latency 14.934 ms 2024-09-21T01:12:59.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 398839 4.28 MB/sec execute 326 sec latency 22.071 ms 2024-09-21T01:12:59.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:12:59 smithi150 ceph-mon[34945]: pgmap v679: 129 pgs: 129 active+clean; 266 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T01:12:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:12:59 smithi160 ceph-mon[34696]: pgmap v679: 129 pgs: 129 active+clean; 266 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T01:12:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:12:59 smithi039 ceph-mon[30415]: pgmap v679: 129 pgs: 129 active+clean; 266 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-09-21T01:13:00.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 400962 4.31 MB/sec execute 327 sec latency 54.165 ms 2024-09-21T01:13:01.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 401150 4.30 MB/sec execute 328 sec latency 72.064 ms 2024-09-21T01:13:01.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:01 smithi150 ceph-mon[34945]: pgmap v680: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-09-21T01:13:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:01 smithi160 ceph-mon[34696]: pgmap v680: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-09-21T01:13:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:01 smithi039 ceph-mon[30415]: pgmap v680: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-09-21T01:13:02.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 402306 4.30 MB/sec execute 329 sec latency 49.316 ms 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:02 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:13:02.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:13:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:02.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:02 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:03.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 402551 4.29 MB/sec execute 330 sec latency 806.635 ms 2024-09-21T01:13:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:03 smithi150 ceph-mon[34945]: pgmap v681: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 154 op/s 2024-09-21T01:13:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:03 smithi160 ceph-mon[34696]: pgmap v681: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 154 op/s 2024-09-21T01:13:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:03 smithi039 ceph-mon[30415]: pgmap v681: 129 pgs: 129 active+clean; 271 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 154 op/s 2024-09-21T01:13:04.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 402551 4.27 MB/sec execute 331 sec latency 1806.676 ms 2024-09-21T01:13:04.863 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:04 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:04.863 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:04 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:04.863 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:04 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:04.863 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:04 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:04 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:04 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:04 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:04 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:04 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:04 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:04 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:04 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 403306 4.27 MB/sec execute 332 sec latency 2001.505 ms 2024-09-21T01:13:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: pgmap v682: 129 pgs: 129 active+clean; 272 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-09-21T01:13:05.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:05.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: osdmap e97: 12 total, 12 up, 12 in 2024-09-21T01:13:05.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:13:05.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:13:05.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:05 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: pgmap v682: 129 pgs: 129 active+clean; 272 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: osdmap e97: 12 total, 12 up, 12 in 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:13:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:05 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: pgmap v682: 129 pgs: 129 active+clean; 272 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-09-21T01:13:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: osdmap e97: 12 total, 12 up, 12 in 2024-09-21T01:13:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:13:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:13:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:05 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:13:06.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 404645 4.29 MB/sec execute 333 sec latency 399.455 ms 2024-09-21T01:13:07.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 405991 4.29 MB/sec execute 334 sec latency 16.350 ms 2024-09-21T01:13:08.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:07 smithi150 ceph-mon[34945]: pgmap v684: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 174 op/s 2024-09-21T01:13:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:07 smithi160 ceph-mon[34696]: pgmap v684: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 174 op/s 2024-09-21T01:13:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:07 smithi039 ceph-mon[30415]: pgmap v684: 129 pgs: 129 active+clean; 274 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 174 op/s 2024-09-21T01:13:08.180 INFO:tasks.workunit.client.0.smithi039.stdout: 1 406092 4.28 MB/sec execute 335 sec latency 934.957 ms 2024-09-21T01:13:09.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 406092 4.27 MB/sec execute 336 sec latency 1935.097 ms 2024-09-21T01:13:10.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:09 smithi039 ceph-mon[30415]: pgmap v685: 129 pgs: 129 active+clean; 277 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-09-21T01:13:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:09 smithi160 ceph-mon[34696]: pgmap v685: 129 pgs: 129 active+clean; 277 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-09-21T01:13:10.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 407119 4.27 MB/sec execute 337 sec latency 2137.057 ms 2024-09-21T01:13:10.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:09 smithi150 ceph-mon[34945]: pgmap v685: 129 pgs: 129 active+clean; 277 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-09-21T01:13:11.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 408063 4.28 MB/sec execute 338 sec latency 678.433 ms 2024-09-21T01:13:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:11 smithi160 ceph-mon[34696]: pgmap v686: 129 pgs: 129 active+clean; 276 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-09-21T01:13:12.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:11 smithi039 ceph-mon[30415]: pgmap v686: 129 pgs: 129 active+clean; 276 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-09-21T01:13:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 409229 4.28 MB/sec execute 339 sec latency 27.658 ms 2024-09-21T01:13:12.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:11 smithi150 ceph-mon[34945]: pgmap v686: 129 pgs: 129 active+clean; 276 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-09-21T01:13:13.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 409641 4.27 MB/sec execute 340 sec latency 749.282 ms 2024-09-21T01:13:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:13 smithi160 ceph-mon[34696]: pgmap v687: 129 pgs: 129 active+clean; 276 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T01:13:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:13 smithi039 ceph-mon[30415]: pgmap v687: 129 pgs: 129 active+clean; 276 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T01:13:14.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 409641 4.26 MB/sec execute 341 sec latency 1749.439 ms 2024-09-21T01:13:14.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:13 smithi150 ceph-mon[34945]: pgmap v687: 129 pgs: 129 active+clean; 276 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-09-21T01:13:15.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 410373 4.25 MB/sec execute 342 sec latency 2129.159 ms 2024-09-21T01:13:15.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:15 smithi150 ceph-mon[34945]: pgmap v688: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 149 op/s 2024-09-21T01:13:15.855 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:15 smithi150 ceph-mon[34945]: osdmap e98: 12 total, 12 up, 12 in 2024-09-21T01:13:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:15 smithi160 ceph-mon[34696]: pgmap v688: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 149 op/s 2024-09-21T01:13:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:15 smithi160 ceph-mon[34696]: osdmap e98: 12 total, 12 up, 12 in 2024-09-21T01:13:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:15 smithi039 ceph-mon[30415]: pgmap v688: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 149 op/s 2024-09-21T01:13:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:15 smithi039 ceph-mon[30415]: osdmap e98: 12 total, 12 up, 12 in 2024-09-21T01:13:16.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 411632 4.27 MB/sec execute 343 sec latency 342.339 ms 2024-09-21T01:13:17.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 412960 4.28 MB/sec execute 344 sec latency 16.294 ms 2024-09-21T01:13:17.673 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:17 smithi160 ceph-mon[34696]: pgmap v690: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 128 op/s 2024-09-21T01:13:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:17 smithi150 ceph-mon[34945]: pgmap v690: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 128 op/s 2024-09-21T01:13:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:17 smithi039 ceph-mon[30415]: pgmap v690: 129 pgs: 129 active+clean; 279 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 128 op/s 2024-09-21T01:13:18.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 413183 4.27 MB/sec execute 345 sec latency 841.997 ms 2024-09-21T01:13:19.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 413183 4.25 MB/sec execute 346 sec latency 1842.125 ms 2024-09-21T01:13:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:19 smithi150 ceph-mon[34945]: pgmap v691: 129 pgs: 129 active+clean; 283 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-09-21T01:13:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:19 smithi160 ceph-mon[34696]: pgmap v691: 129 pgs: 129 active+clean; 283 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-09-21T01:13:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:19 smithi039 ceph-mon[30415]: pgmap v691: 129 pgs: 129 active+clean; 283 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-09-21T01:13:20.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 414514 4.26 MB/sec execute 347 sec latency 2044.787 ms 2024-09-21T01:13:21.181 INFO:tasks.workunit.client.0.smithi039.stdout: 1 414699 4.25 MB/sec execute 348 sec latency 1171.183 ms 2024-09-21T01:13:21.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:21 smithi150 ceph-mon[34945]: pgmap v692: 129 pgs: 129 active+clean; 286 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-09-21T01:13:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:21 smithi160 ceph-mon[34696]: pgmap v692: 129 pgs: 129 active+clean; 286 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-09-21T01:13:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:21 smithi039 ceph-mon[30415]: pgmap v692: 129 pgs: 129 active+clean; 286 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-09-21T01:13:22.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 416021 4.26 MB/sec execute 349 sec latency 17.370 ms 2024-09-21T01:13:23.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 416732 4.26 MB/sec execute 350 sec latency 554.210 ms 2024-09-21T01:13:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:23 smithi150 ceph-mon[34945]: pgmap v693: 129 pgs: 129 active+clean; 287 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 138 op/s 2024-09-21T01:13:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:23 smithi160 ceph-mon[34696]: pgmap v693: 129 pgs: 129 active+clean; 287 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 138 op/s 2024-09-21T01:13:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:23 smithi039 ceph-mon[30415]: pgmap v693: 129 pgs: 129 active+clean; 287 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 138 op/s 2024-09-21T01:13:24.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 416732 4.25 MB/sec execute 351 sec latency 1554.365 ms 2024-09-21T01:13:25.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 418038 4.26 MB/sec execute 352 sec latency 1908.236 ms 2024-09-21T01:13:25.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:25 smithi039 ceph-mon[30415]: pgmap v694: 129 pgs: 129 active+clean; 287 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 119 op/s 2024-09-21T01:13:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:25 smithi150 ceph-mon[34945]: pgmap v694: 129 pgs: 129 active+clean; 287 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 119 op/s 2024-09-21T01:13:25.858 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:25 smithi160 ceph-mon[34696]: pgmap v694: 129 pgs: 129 active+clean; 287 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 119 op/s 2024-09-21T01:13:26.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 418673 4.26 MB/sec execute 353 sec latency 719.449 ms 2024-09-21T01:13:27.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 419984 4.27 MB/sec execute 354 sec latency 15.954 ms 2024-09-21T01:13:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:27 smithi150 ceph-mon[34945]: pgmap v695: 129 pgs: 129 active+clean; 289 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-09-21T01:13:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:27 smithi160 ceph-mon[34696]: pgmap v695: 129 pgs: 129 active+clean; 289 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-09-21T01:13:27.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:27 smithi039 ceph-mon[30415]: pgmap v695: 129 pgs: 129 active+clean; 289 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-09-21T01:13:28.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 420275 4.25 MB/sec execute 355 sec latency 632.383 ms 2024-09-21T01:13:29.182 INFO:tasks.workunit.client.0.smithi039.stdout: 1 420275 4.24 MB/sec execute 356 sec latency 1632.515 ms 2024-09-21T01:13:29.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:29 smithi150 ceph-mon[34945]: pgmap v696: 129 pgs: 129 active+clean; 289 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 139 op/s 2024-09-21T01:13:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:29 smithi160 ceph-mon[34696]: pgmap v696: 129 pgs: 129 active+clean; 289 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 139 op/s 2024-09-21T01:13:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:29 smithi039 ceph-mon[30415]: pgmap v696: 129 pgs: 129 active+clean; 289 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 139 op/s 2024-09-21T01:13:30.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 421769 4.25 MB/sec execute 357 sec latency 1833.147 ms 2024-09-21T01:13:31.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 421777 4.24 MB/sec execute 358 sec latency 996.726 ms 2024-09-21T01:13:31.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:31 smithi150 ceph-mon[34945]: pgmap v697: 129 pgs: 129 active+clean; 292 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-09-21T01:13:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:31 smithi160 ceph-mon[34696]: pgmap v697: 129 pgs: 129 active+clean; 292 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-09-21T01:13:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:31 smithi039 ceph-mon[30415]: pgmap v697: 129 pgs: 129 active+clean; 292 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 122 op/s 2024-09-21T01:13:32.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 423003 4.25 MB/sec execute 359 sec latency 1027.971 ms 2024-09-21T01:13:33.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 423824 4.25 MB/sec execute 360 sec latency 474.857 ms 2024-09-21T01:13:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:33 smithi150 ceph-mon[34945]: pgmap v698: 129 pgs: 129 active+clean; 293 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-09-21T01:13:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:33 smithi160 ceph-mon[34696]: pgmap v698: 129 pgs: 129 active+clean; 293 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-09-21T01:13:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:33 smithi039 ceph-mon[30415]: pgmap v698: 129 pgs: 129 active+clean; 293 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-09-21T01:13:34.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 423824 4.24 MB/sec execute 361 sec latency 1474.946 ms 2024-09-21T01:13:35.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 425737 4.26 MB/sec execute 362 sec latency 1668.052 ms 2024-09-21T01:13:35.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:35 smithi039 ceph-mon[30415]: pgmap v699: 129 pgs: 129 active+clean; 293 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 118 op/s 2024-09-21T01:13:35.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:35 smithi150 ceph-mon[34945]: pgmap v699: 129 pgs: 129 active+clean; 293 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 118 op/s 2024-09-21T01:13:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:35 smithi160 ceph-mon[34696]: pgmap v699: 129 pgs: 129 active+clean; 293 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 118 op/s 2024-09-21T01:13:36.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 426340 4.25 MB/sec execute 363 sec latency 278.759 ms 2024-09-21T01:13:37.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 427365 4.25 MB/sec execute 364 sec latency 108.634 ms 2024-09-21T01:13:37.714 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:13:37.714 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-21T01:13:37.715 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:13:37.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:37 smithi039 ceph-mon[30415]: pgmap v700: 129 pgs: 129 active+clean; 297 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 131 op/s 2024-09-21T01:13:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:37 smithi150 ceph-mon[34945]: pgmap v700: 129 pgs: 129 active+clean; 297 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 131 op/s 2024-09-21T01:13:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:37 smithi160 ceph-mon[34696]: pgmap v700: 129 pgs: 129 active+clean; 297 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 131 op/s 2024-09-21T01:13:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:13:38.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 427365 4.24 MB/sec execute 365 sec latency 1108.747 ms 2024-09-21T01:13:38.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.877+0000 7fc6ca800640 1 -- 172.21.15.39:0/46430060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 msgr2=0x7fc6c41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:38.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.877+0000 7fc6ca800640 1 --2- 172.21.15.39:0/46430060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c41098c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009900 tx=0x7fc6b002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 -- 172.21.15.39:0/46430060 shutdown_connections 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 --2- 172.21.15.39:0/46430060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c4109e00 0x7fc6c4110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 --2- 172.21.15.39:0/46430060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c41098c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 --2- 172.21.15.39:0/46430060 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4103b70 0x7fc6c4103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 -- 172.21.15.39:0/46430060 >> 172.21.15.39:0/46430060 conn(0x7fc6c40ff5e0 msgr2=0x7fc6c4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 -- 172.21.15.39:0/46430060 shutdown_connections 2024-09-21T01:13:38.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.878+0000 7fc6ca800640 1 -- 172.21.15.39:0/46430060 wait complete. 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 Processor -- start 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 -- start start 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c4103b70 0x7fc6c4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4109e00 0x7fc6c4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4113b10 con 0x7fc6c41094b0 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4113990 con 0x7fc6c4103b70 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.879+0000 7fc6ca800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4113c90 con 0x7fc6c4109e00 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c4075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60116/0 (socket says 172.21.15.39:60116) 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 -- 172.21.15.39:0/4106424235 learned_addr learned my addr 172.21.15.39:0/4106424235 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c3400640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c4103b70 0x7fc6c4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c3e00640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4109e00 0x7fc6c4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 -- 172.21.15.39:0/4106424235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4109e00 msgr2=0x7fc6c4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4109e00 0x7fc6c4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 -- 172.21.15.39:0/4106424235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c4103b70 msgr2=0x7fc6c4075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c4103b70 0x7fc6c4075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c2a00640 1 -- 172.21.15.39:0/4106424235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c41a8760 con 0x7fc6c41094b0 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.880+0000 7fc6c3e00640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4109e00 0x7fc6c4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:13:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.881+0000 7fc6c2a00640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c4075c50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0005ab0 tx=0x7fc6b0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:38.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.881+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b003f070 con 0x7fc6c41094b0 2024-09-21T01:13:38.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.881+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6b0031b50 con 0x7fc6c41094b0 2024-09-21T01:13:38.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.881+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c41a89f0 con 0x7fc6c41094b0 2024-09-21T01:13:38.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.881+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0035030 con 0x7fc6c41094b0 2024-09-21T01:13:38.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.881+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c41a8ed0 con 0x7fc6c41094b0 2024-09-21T01:13:38.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.882+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c4103f80 con 0x7fc6c41094b0 2024-09-21T01:13:38.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.885+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fc6b0042070 con 0x7fc6c41094b0 2024-09-21T01:13:38.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.886+0000 7fc6bbe00640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc6a80b6680 0x7fc6a80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:38.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.887+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7fc6b0108500 con 0x7fc6c41094b0 2024-09-21T01:13:38.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.887+0000 7fc6c3400640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc6a80b6680 0x7fc6a80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:38.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.887+0000 7fc6c3400640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc6a80b6680 0x7fc6a80b8b90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4077160 tx=0x7fc6b4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:38.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:38.889+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc6b00d29f0 con 0x7fc6c41094b0 2024-09-21T01:13:39.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.126+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6c402e640 con 0x7fc6c41094b0 2024-09-21T01:13:39.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.127+0000 7fc6bbe00640 1 -- 172.21.15.39:0/4106424235 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11917 (secure 0 0 0) 0x7fc6b00d7930 con 0x7fc6c41094b0 2024-09-21T01:13:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:13:39.127 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"btime":"2024-09-21T01:12:25:429703+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:12:24.519635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:13:39.128 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc6a80b6680 msgr2=0x7fc6a80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc6a80b6680 0x7fc6a80b8b90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4077160 tx=0x7fc6b4005f90 comp rx=0 tx=0).stop 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 msgr2=0x7fc6c4075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c4075c50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0005ab0 tx=0x7fc6b0003b90 comp rx=0 tx=0).stop 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 shutdown_connections 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc6c4109e00 0x7fc6c4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c41094b0 0x7fc6c4075c50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc6a80b6680 0x7fc6a80b8b90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 --2- 172.21.15.39:0/4106424235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c4103b70 0x7fc6c4075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 >> 172.21.15.39:0/4106424235 conn(0x7fc6c40ff5e0 msgr2=0x7fc6c4101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.129+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 shutdown_connections 2024-09-21T01:13:39.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:39.130+0000 7fc6ca800640 1 -- 172.21.15.39:0/4106424235 wait complete. 2024-09-21T01:13:39.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 427365 4.23 MB/sec execute 366 sec latency 2108.808 ms 2024-09-21T01:13:39.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:39 smithi150 ceph-mon[34945]: pgmap v701: 129 pgs: 129 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-09-21T01:13:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:39 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4106424235' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:13:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:39 smithi160 ceph-mon[34696]: pgmap v701: 129 pgs: 129 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-09-21T01:13:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:39 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4106424235' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:13:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:39 smithi039 ceph-mon[30415]: pgmap v701: 129 pgs: 129 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-09-21T01:13:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:39 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4106424235' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:13:40.137 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-21T01:13:40.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-21T01:13:40.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 429282 4.25 MB/sec execute 367 sec latency 2307.942 ms 2024-09-21T01:13:40.602 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:13:41.183 INFO:tasks.workunit.client.0.smithi039.stdout: 1 429684 4.25 MB/sec execute 368 sec latency 446.013 ms 2024-09-21T01:13:41.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 -- 172.21.15.39:0/177801749 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d810dc70 msgr2=0x7f51d8110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 --2- 172.21.15.39:0/177801749 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d810dc70 0x7f51d8110140 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009ca0 tx=0x7f51d402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 -- 172.21.15.39:0/177801749 shutdown_connections 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 --2- 172.21.15.39:0/177801749 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d810dc70 0x7f51d8110140 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 --2- 172.21.15.39:0/177801749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51d81050f0 0x7f51d810d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 --2- 172.21.15.39:0/177801749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81046c0 0x7f51d8104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 -- 172.21.15.39:0/177801749 >> 172.21.15.39:0/177801749 conn(0x7f51d80fe330 msgr2=0x7f51d8100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 -- 172.21.15.39:0/177801749 shutdown_connections 2024-09-21T01:13:41.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.462+0000 7f51df200640 1 -- 172.21.15.39:0/177801749 wait complete. 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.463+0000 7f51df200640 1 Processor -- start 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.463+0000 7f51df200640 1 -- start start 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51df200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51d81046c0 0x7f51d819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51df200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 0x7f51d819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51df200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d810dc70 0x7f51d819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51df200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8113900 con 0x7f51d810dc70 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51df200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8113780 con 0x7f51d81046c0 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51df200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8113a80 con 0x7f51d81050f0 2024-09-21T01:13:41.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 0x7f51d819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 0x7f51d819a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39194/0 (socket says 172.21.15.39:39194) 2024-09-21T01:13:41.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 -- 172.21.15.39:0/1775704376 learned_addr learned my addr 172.21.15.39:0/1775704376 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:13:41.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dde00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51d81046c0 0x7f51d819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51de800640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d810dc70 0x7f51d819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 -- 172.21.15.39:0/1775704376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51d81046c0 msgr2=0x7f51d819a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51d81046c0 0x7f51d819a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 -- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d810dc70 msgr2=0x7f51d819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d810dc70 0x7f51d819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.464+0000 7f51dd400640 1 -- 172.21.15.39:0/1775704376 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d81a0250 con 0x7f51d81050f0 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.465+0000 7f51dd400640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 0x7f51d819a880 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51c800e9f0 tx=0x7f51c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.465+0000 7f51d2a00640 1 -- 172.21.15.39:0/1775704376 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c800ccc0 con 0x7f51d81050f0 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.465+0000 7f51d2a00640 1 -- 172.21.15.39:0/1775704376 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c80054d0 con 0x7f51d81050f0 2024-09-21T01:13:41.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.465+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d81a0540 con 0x7f51d81050f0 2024-09-21T01:13:41.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.465+0000 7f51d2a00640 1 -- 172.21.15.39:0/1775704376 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8010430 con 0x7f51d81050f0 2024-09-21T01:13:41.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.465+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d81a6ca0 con 0x7f51d81050f0 2024-09-21T01:13:41.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.469+0000 7f51e7473740 1 Processor -- start 2024-09-21T01:13:41.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.469+0000 7f51e7473740 1 -- start start 2024-09-21T01:13:41.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.469+0000 7f51d2a00640 1 -- 172.21.15.39:0/1775704376 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f51c80046c0 con 0x7f51d81050f0 2024-09-21T01:13:41.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.469+0000 7f51e7473740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfbb5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.469+0000 7f51e7473740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7e4980 0x55d5dfbb5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51e7473740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51e7473740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d5dfb2aca0 con 0x55d5df7e4980 2024-09-21T01:13:41.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51e7473740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d5dfb1a0d0 con 0x55d5df7ecb60 2024-09-21T01:13:41.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51e7473740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d5dfb1dee0 con 0x55d5df80d840 2024-09-21T01:13:41.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfbb5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfbb5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39206/0 (socket says 172.21.15.39:39206) 2024-09-21T01:13:41.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 -- 172.21.15.39:0/2729070487 learned_addr learned my addr 172.21.15.39:0/2729070487 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:13:41.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51dd400640 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51dde00640 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7e4980 0x55d5dfbb5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 -- 172.21.15.39:0/2729070487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7ecb60 msgr2=0x55d5dfbb5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 -- 172.21.15.39:0/2729070487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7e4980 msgr2=0x55d5dfbb5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.470+0000 7f51de800640 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7e4980 0x55d5dfbb5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.471+0000 7f51de800640 1 -- 172.21.15.39:0/2729070487 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5dfacc400 con 0x55d5df80d840 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.471+0000 7f51de800640 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfbb5510 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009ca0 tx=0x7f51d402f310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2527fc6e33edab1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.471+0000 7f51c7000640 1 -- 172.21.15.39:0/2729070487 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4031db0 con 0x55d5df80d840 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.471+0000 7f51c7000640 1 -- 172.21.15.39:0/2729070487 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d4036030 con 0x55d5df80d840 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.471+0000 7f51c7000640 1 -- 172.21.15.39:0/2729070487 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4004330 con 0x55d5df80d840 2024-09-21T01:13:41.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51d2a00640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51c00b66f0 0x7f51c00b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51d2a00640 1 -- 172.21.15.39:0/1775704376 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7f51c80e7e10 con 0x7f51d81050f0 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51dde00640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51c00b66f0 0x7f51c00b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 -- 172.21.15.39:0/2729070487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 msgr2=0x55d5dfbb5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfbb5510 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009ca0 tx=0x7f51d402f310 comp rx=0 tx=0).stop 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 -- 172.21.15.39:0/2729070487 shutdown_connections 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7e4980 0x55d5dfbb5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 --2- 172.21.15.39:0/2729070487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfbb5510 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.472+0000 7f51e7473740 1 -- 172.21.15.39:0/2729070487 >> 172.21.15.39:0/2729070487 conn(0x55d5df801990 msgr2=0x55d5dfbb3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.473+0000 7f51e7473740 1 -- 172.21.15.39:0/2729070487 shutdown_connections 2024-09-21T01:13:41.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.473+0000 7f51e7473740 1 -- 172.21.15.39:0/2729070487 wait complete. 2024-09-21T01:13:41.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.473+0000 7f51dde00640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51c00b66f0 0x7f51c00b8c00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55d5dfbb4190 tx=0x7f51cc00b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.473+0000 7f51e7473740 1 -- start start 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 Processor -- start 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d5df97b7a0 con 0x55d5df7ecb60 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d5dfabfa10 con 0x55d5df80d840 2024-09-21T01:13:41.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d5dfbc7f20 con 0x55d5df7e4980 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51e7473740 10 client.0 _ll_register_callbacks cb 0x55d5dfacfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51dde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51dd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60164/0 (socket says 172.21.15.39:60164) 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.474+0000 7f51dde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:39212/0 (socket says 172.21.15.39:39212) 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dd400640 1 -- 172.21.15.39:0/239188013 learned_addr learned my addr 172.21.15.39:0/239188013 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51de800640 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dde00640 1 -- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 msgr2=0x55d5dfbd1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dde00640 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dde00640 1 -- 172.21.15.39:0/239188013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 msgr2=0x55d5dfbc1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dde00640 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dde00640 1 -- 172.21.15.39:0/239188013 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5df887b90 con 0x55d5df7e4980 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51dde00640 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc0be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51cc005d00 tx=0x7f51cc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a85777f76bb2636 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:41.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.475+0000 7f51bb400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:13:41.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.476+0000 7f51bb400640 1 -- 172.21.15.39:0/239188013 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc02c060 con 0x55d5df7e4980 2024-09-21T01:13:41.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.476+0000 7f51bb400640 1 -- 172.21.15.39:0/239188013 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7f51cc017030 con 0x55d5df7e4980 2024-09-21T01:13:41.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.476+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d4036030 con 0x55d5df7e4980 2024-09-21T01:13:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.485+0000 7f51bb400640 1 -- 172.21.15.39:0/239188013 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f51cc033050 con 0x55d5df7e4980 2024-09-21T01:13:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.485+0000 7f51e7473740 10 client.25933 fetch_fsmap learned FSMap version 34 2024-09-21T01:13:41.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.485+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d5dfbd3c80 con 0x55d5df7e4980 2024-09-21T01:13:41.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.485+0000 7f51bb400640 1 -- 172.21.15.39:0/239188013 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d4036030 con 0x55d5df7e4980 2024-09-21T01:13:41.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51bb400640 1 -- 172.21.15.39:0/239188013 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f51cc0107d0 con 0x55d5df7e4980 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51e7473740 10 client.25933 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51e7473740 10 client.25933 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51e7473740 20 client.25933 populate_metadata read hostname 'smithi039' 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51e7473740 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7cc1c0 0x55d5dfbd7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51e7473740 4 client.25933 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5dfb2cbf0 con 0x55d5df7cc1c0 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.486+0000 7f51dd400640 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7cc1c0 0x55d5dfbd7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.487+0000 7f51dd400640 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7cc1c0 0x55d5dfbd7220 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ace3a8e87d15f0 server_cookie=de8e554daacf0753 in_seq=0 out_seq=0 2024-09-21T01:13:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.487+0000 7f51bb400640 10 client.25933 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:13:41.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.487+0000 7f51bb400640 1 -- 172.21.15.39:0/239188013 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d5dfb2cbf0 con 0x55d5df7cc1c0 2024-09-21T01:13:41.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.487+0000 7f51bb400640 10 client.25933 handle_command_reply: tid=0 2024-09-21T01:13:41.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.487+0000 7f51e7473740 1 client.25933 shutdown 2024-09-21T01:13:41.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 msgr2=0x55d5dfbc0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc0be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51cc005d00 tx=0x7f51cc004730 comp rx=0 tx=0).stop 2024-09-21T01:13:41.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 shutdown_connections 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7cc1c0 0x55d5dfbd7220 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51bb400640 0 client.25933 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 --2- 172.21.15.39:0/239188013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc0be0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 >> 172.21.15.39:0/239188013 conn(0x55d5df801990 msgr2=0x55d5dfbb86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 shutdown_connections 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 1 -- 172.21.15.39:0/239188013 wait complete. 2024-09-21T01:13:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.488+0000 7f51e7473740 20 client.25933 trim_cache size 0 max 16384 2024-09-21T01:13:41.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.508+0000 7f51e7473740 1 Processor -- start 2024-09-21T01:13:41.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.508+0000 7f51e7473740 1 -- start start 2024-09-21T01:13:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51e7473740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51e7473740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51e7473740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51e7473740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81b14e0 con 0x55d5df7ecb60 2024-09-21T01:13:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51e7473740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81aed50 con 0x55d5df80d840 2024-09-21T01:13:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51e7473740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81acbc0 con 0x55d5df7e4980 2024-09-21T01:13:41.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51de800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51dde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51dd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51de800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38148/0 (socket says 172.21.15.39:38148) 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.509+0000 7f51de800640 1 -- 172.21.15.39:0/243118530 learned_addr learned my addr 172.21.15.39:0/243118530 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51de800640 1 -- 172.21.15.39:0/243118530 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 msgr2=0x55d5dfbc2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51de800640 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51de800640 1 -- 172.21.15.39:0/243118530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 msgr2=0x55d5dfbb6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51de800640 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51de800640 1 -- 172.21.15.39:0/243118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5dfbaebf0 con 0x55d5df80d840 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51dde00640 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51dd400640 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51de800640 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1140 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51d4003e50 tx=0x7f51d4003de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b038ecf991baee92 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.510+0000 7f51c6600640 1 -- 172.21.15.39:0/243118530 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4056070 con 0x55d5df80d840 2024-09-21T01:13:41.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.511+0000 7f51c6600640 1 -- 172.21.15.39:0/243118530 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d4036040 con 0x55d5df80d840 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.511+0000 7f51c6600640 1 -- 172.21.15.39:0/243118530 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d404a040 con 0x55d5df80d840 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.511+0000 7f51e7473740 1 -- 172.21.15.39:0/243118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 msgr2=0x55d5dfbd1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.511+0000 7f51e7473740 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1140 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51d4003e50 tx=0x7f51d4003de0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 -- 172.21.15.39:0/243118530 shutdown_connections 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df80d840 0x55d5dfbd1140 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbb6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 --2- 172.21.15.39:0/243118530 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df7e4980 0x55d5dfbc2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 -- 172.21.15.39:0/243118530 >> 172.21.15.39:0/243118530 conn(0x55d5df801990 msgr2=0x55d5dfbb6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:41.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 -- 172.21.15.39:0/243118530 shutdown_connections 2024-09-21T01:13:41.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 -- 172.21.15.39:0/243118530 wait complete. 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.512+0000 7f51e7473740 1 -- start start 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 Processor -- start 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 0x55d5dfbc9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfc40be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ad210 con 0x55d5df7ecb60 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ae680 con 0x55d5df7e4980 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ad8e0 con 0x55d5df80d840 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.513+0000 7f51e7473740 10 client.0 _ll_register_callbacks cb 0x7f51cc00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 0x55d5dfbc9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51de800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfc40be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 0x55d5dfbc9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38164/0 (socket says 172.21.15.39:38164) 2024-09-21T01:13:41.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 -- 172.21.15.39:0/1318036013 learned_addr learned my addr 172.21.15.39:0/1318036013 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:13:41.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 -- 172.21.15.39:0/1318036013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 msgr2=0x55d5dfc40be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfc40be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 -- 172.21.15.39:0/1318036013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 msgr2=0x55d5dfbc9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:13:41.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 -- 172.21.15.39:0/1318036013 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5dfc41120 con 0x55d5df7e4980 2024-09-21T01:13:41.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51dde00640 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 0x55d5dfbc9530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f51cc010ba0 tx=0x7f51cc010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22ee29c57838cab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.514+0000 7f51bb400640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51bb400640 1 -- 172.21.15.39:0/1318036013 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc01f220 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51bb400640 1 -- 172.21.15.39:0/1318036013 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7f51cc010e10 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d4036040 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51bb400640 1 -- 172.21.15.39:0/1318036013 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f51cc005030 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51e7473740 10 client.25949 fetch_fsmap learned FSMap version 34 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d5dfc421f0 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.515+0000 7f51bb400640 1 -- 172.21.15.39:0/1318036013 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d4036040 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51bb400640 1 -- 172.21.15.39:0/1318036013 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f51cc004a10 con 0x55d5df7e4980 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51e7473740 10 client.25949 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51e7473740 10 client.25949 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51e7473740 20 client.25949 populate_metadata read hostname 'smithi039' 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51e7473740 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7a1570 0x55d5dfc44aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51e7473740 4 client.25949 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-21T01:13:41.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x55d5dfb286a0 con 0x55d5df7a1570 2024-09-21T01:13:41.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.516+0000 7f51dd400640 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7a1570 0x55d5dfc44aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:13:41.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.517+0000 7f51dd400640 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7a1570 0x55d5dfc44aa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27373d561a5b6862 server_cookie=8e79335cb4538bec in_seq=0 out_seq=0 2024-09-21T01:13:41.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.517+0000 7f51bb400640 10 client.25949 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51bb400640 1 -- 172.21.15.39:0/1318036013 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d5dfb286a0 con 0x55d5df7a1570 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51bb400640 10 client.25949 handle_command_reply: tid=0 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 client.25949 shutdown 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 msgr2=0x55d5dfbc9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 0x55d5dfbc9530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f51cc010ba0 tx=0x7f51cc010660 comp rx=0 tx=0).stop 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 shutdown_connections 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d5df7a1570 0x55d5dfc44aa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d5df80d840 0x55d5dfc40be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5df7ecb60 0x55d5dfbc9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 --2- 172.21.15.39:0/1318036013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d5df7e4980 0x55d5dfbc9530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51bb400640 0 client.25949 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 >> 172.21.15.39:0/1318036013 conn(0x55d5df801990 msgr2=0x55d5dfbc2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 shutdown_connections 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.572+0000 7f51e7473740 1 -- 172.21.15.39:0/1318036013 wait complete. 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51e7473740 20 client.25949 trim_cache size 0 max 16384 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 34, 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 38, 2024-09-21T01:13:41.574 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "ddad0d1": { 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 38, 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 144.5, 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:13:41.575 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51c00b66f0 msgr2=0x7f51c00b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51c00b66f0 0x7f51c00b8c00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55d5dfbb4190 tx=0x7f51cc00b410 comp rx=0 tx=0).stop 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 msgr2=0x7f51d819a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 0x7f51d819a880 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51c800e9f0 tx=0x7f51c800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 shutdown_connections 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d810dc70 0x7f51d819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f51d81050f0 0x7f51d819a880 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f51c00b66f0 0x7f51c00b8c00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 --2- 172.21.15.39:0/1775704376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51d81046c0 0x7f51d819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.573+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 >> 172.21.15.39:0/1775704376 conn(0x7f51d80fe330 msgr2=0x7f51d8107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.574+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 shutdown_connections 2024-09-21T01:13:41.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:13:41.574+0000 7f51df200640 1 -- 172.21.15.39:0/1775704376 wait complete. 2024-09-21T01:13:41.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:41 smithi039 ceph-mon[30415]: pgmap v702: 129 pgs: 129 active+clean; 299 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:13:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:41 smithi150 ceph-mon[34945]: pgmap v702: 129 pgs: 129 active+clean; 299 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:13:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:41 smithi160 ceph-mon[34696]: pgmap v702: 129 pgs: 129 active+clean; 299 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:13:42.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.24 MB/sec execute 369 sec latency 643.144 ms 2024-09-21T01:13:42.483 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ddad0d1', quiesce took 4.8 seconds. Will release after: 19.7 2024-09-21T01:13:43.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.22 MB/sec execute 370 sec latency 1643.315 ms 2024-09-21T01:13:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:43 smithi150 ceph-mon[34945]: pgmap v703: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-09-21T01:13:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:43 smithi160 ceph-mon[34696]: pgmap v703: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-09-21T01:13:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:43 smithi039 ceph-mon[30415]: pgmap v703: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-09-21T01:13:44.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.21 MB/sec execute 371 sec latency 2643.457 ms 2024-09-21T01:13:45.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.20 MB/sec execute 372 sec latency 3643.599 ms 2024-09-21T01:13:45.752 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:45 smithi039 ceph-mon[30415]: pgmap v704: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-09-21T01:13:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:45 smithi150 ceph-mon[34945]: pgmap v704: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-09-21T01:13:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:45 smithi160 ceph-mon[34696]: pgmap v704: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-09-21T01:13:46.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.19 MB/sec execute 373 sec latency 4643.703 ms 2024-09-21T01:13:47.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.18 MB/sec execute 374 sec latency 5643.771 ms 2024-09-21T01:13:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:47 smithi150 ceph-mon[34945]: pgmap v705: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T01:13:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:47 smithi160 ceph-mon[34696]: pgmap v705: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T01:13:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:47 smithi039 ceph-mon[30415]: pgmap v705: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-09-21T01:13:48.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.17 MB/sec execute 375 sec latency 6643.934 ms 2024-09-21T01:13:49.184 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.16 MB/sec execute 376 sec latency 7644.078 ms 2024-09-21T01:13:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:49 smithi150 ceph-mon[34945]: pgmap v706: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 1.2 MiB/s wr, 102 op/s 2024-09-21T01:13:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:49 smithi160 ceph-mon[34696]: pgmap v706: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 1.2 MiB/s wr, 102 op/s 2024-09-21T01:13:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:49 smithi039 ceph-mon[30415]: pgmap v706: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 1.2 MiB/s wr, 102 op/s 2024-09-21T01:13:50.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.15 MB/sec execute 377 sec latency 8644.227 ms 2024-09-21T01:13:51.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.13 MB/sec execute 378 sec latency 9644.335 ms 2024-09-21T01:13:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:51 smithi150 ceph-mon[34945]: pgmap v707: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.0 MiB/s wr, 72 op/s 2024-09-21T01:13:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:51 smithi160 ceph-mon[34696]: pgmap v707: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.0 MiB/s wr, 72 op/s 2024-09-21T01:13:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:51 smithi039 ceph-mon[30415]: pgmap v707: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.0 MiB/s wr, 72 op/s 2024-09-21T01:13:52.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.12 MB/sec execute 379 sec latency 10644.380 ms 2024-09-21T01:13:53.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.11 MB/sec execute 380 sec latency 11644.524 ms 2024-09-21T01:13:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:53 smithi150 ceph-mon[34945]: pgmap v708: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.0 MiB/s wr, 64 op/s 2024-09-21T01:13:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:53 smithi160 ceph-mon[34696]: pgmap v708: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.0 MiB/s wr, 64 op/s 2024-09-21T01:13:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:53 smithi039 ceph-mon[30415]: pgmap v708: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 1.0 MiB/s wr, 64 op/s 2024-09-21T01:13:54.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.10 MB/sec execute 381 sec latency 12644.671 ms 2024-09-21T01:13:55.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.09 MB/sec execute 382 sec latency 13644.723 ms 2024-09-21T01:13:55.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:55 smithi039 ceph-mon[30415]: pgmap v709: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 790 KiB/s wr, 44 op/s 2024-09-21T01:13:55.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:55 smithi150 ceph-mon[34945]: pgmap v709: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 790 KiB/s wr, 44 op/s 2024-09-21T01:13:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:55 smithi160 ceph-mon[34696]: pgmap v709: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 790 KiB/s wr, 44 op/s 2024-09-21T01:13:56.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.08 MB/sec execute 383 sec latency 14644.772 ms 2024-09-21T01:13:57.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.07 MB/sec execute 384 sec latency 15644.884 ms 2024-09-21T01:13:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:57 smithi150 ceph-mon[34945]: pgmap v710: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 742 KiB/s wr, 30 op/s 2024-09-21T01:13:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:57 smithi160 ceph-mon[34696]: pgmap v710: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 742 KiB/s wr, 30 op/s 2024-09-21T01:13:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:57 smithi039 ceph-mon[30415]: pgmap v710: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 742 KiB/s wr, 30 op/s 2024-09-21T01:13:58.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.06 MB/sec execute 385 sec latency 16645.046 ms 2024-09-21T01:13:59.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.05 MB/sec execute 386 sec latency 17645.203 ms 2024-09-21T01:13:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:13:59 smithi150 ceph-mon[34945]: pgmap v711: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 882 KiB/s wr, 37 op/s 2024-09-21T01:13:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:13:59 smithi160 ceph-mon[34696]: pgmap v711: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 882 KiB/s wr, 37 op/s 2024-09-21T01:13:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:13:59 smithi039 ceph-mon[30415]: pgmap v711: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 882 KiB/s wr, 37 op/s 2024-09-21T01:14:00.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.04 MB/sec execute 387 sec latency 18645.293 ms 2024-09-21T01:14:01.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.03 MB/sec execute 388 sec latency 19645.365 ms 2024-09-21T01:14:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:01 smithi150 ceph-mon[34945]: pgmap v712: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 728 KiB/s wr, 30 op/s 2024-09-21T01:14:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:01 smithi160 ceph-mon[34696]: pgmap v712: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 728 KiB/s wr, 30 op/s 2024-09-21T01:14:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:01 smithi039 ceph-mon[30415]: pgmap v712: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 728 KiB/s wr, 30 op/s 2024-09-21T01:14:02.184 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:14:02.185 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.02 MB/sec execute 389 sec latency 20645.435 ms 2024-09-21T01:14:02.585 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:14:03.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.01 MB/sec execute 390 sec latency 21645.490 ms 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 -- 172.21.15.39:0/101441887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff864109df0 msgr2=0x7ff864110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 --2- 172.21.15.39:0/101441887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff864109df0 0x7ff864110620 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009ca0 tx=0x7ff85c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 -- 172.21.15.39:0/101441887 shutdown_connections 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 --2- 172.21.15.39:0/101441887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff864109df0 0x7ff864110620 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 --2- 172.21.15.39:0/101441887 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff8641094a0 0x7ff8641098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 --2- 172.21.15.39:0/101441887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 0x7ff864103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 -- 172.21.15.39:0/101441887 >> 172.21.15.39:0/101441887 conn(0x7ff86407b850 msgr2=0x7ff86407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 -- 172.21.15.39:0/101441887 shutdown_connections 2024-09-21T01:14:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.326+0000 7ff869800640 1 -- 172.21.15.39:0/101441887 wait complete. 2024-09-21T01:14:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.327+0000 7ff869800640 1 Processor -- start 2024-09-21T01:14:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.327+0000 7ff869800640 1 -- start start 2024-09-21T01:14:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff869800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 0x7ff864075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff869800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 0x7ff864075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff869800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff864109df0 0x7ff864076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff869800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff864113b00 con 0x7ff8641094a0 2024-09-21T01:14:03.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff869800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff864113980 con 0x7ff864103b60 2024-09-21T01:14:03.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff869800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff864113c80 con 0x7ff864109df0 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff862000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 0x7ff864075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff862a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 0x7ff864075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff862a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 0x7ff864075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50352/0 (socket says 172.21.15.39:50352) 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff862000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 0x7ff864075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51454/0 (socket says 172.21.15.39:51454) 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff862a00640 1 -- 172.21.15.39:0/4054101158 learned_addr learned my addr 172.21.15.39:0/4054101158 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.328+0000 7ff863400640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff864109df0 0x7ff864076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff862000640 1 -- 172.21.15.39:0/4054101158 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff864109df0 msgr2=0x7ff864076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff862000640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff864109df0 0x7ff864076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff862000640 1 -- 172.21.15.39:0/4054101158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 msgr2=0x7ff864075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff862000640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 0x7ff864075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff862000640 1 -- 172.21.15.39:0/4054101158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8641a86d0 con 0x7ff8641094a0 2024-09-21T01:14:03.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff862000640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 0x7ff864075c50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff86406c9f0 tx=0x7ff85000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:03.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85000cc40 con 0x7ff8641094a0 2024-09-21T01:14:03.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.329+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8641a89c0 con 0x7ff8641094a0 2024-09-21T01:14:03.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.330+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85000cde0 con 0x7ff8641094a0 2024-09-21T01:14:03.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.330+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850004a00 con 0x7ff8641094a0 2024-09-21T01:14:03.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.330+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8641a8f00 con 0x7ff8641094a0 2024-09-21T01:14:03.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.331+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff864103f70 con 0x7ff8641094a0 2024-09-21T01:14:03.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.333+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff850004ba0 con 0x7ff8641094a0 2024-09-21T01:14:03.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.335+0000 7ff85b400640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8480b6680 0x7ff8480b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:03.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.335+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7ff8500e7bc0 con 0x7ff8641094a0 2024-09-21T01:14:03.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.335+0000 7ff862a00640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8480b6680 0x7ff8480b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:03.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.336+0000 7ff862a00640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8480b6680 0x7ff8480b8b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff864077160 tx=0x7ff854005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:03.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.337+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff8500add60 con 0x7ff8641094a0 2024-09-21T01:14:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.573+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff864076a10 con 0x7ff8641094a0 2024-09-21T01:14:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.574+0000 7ff85b400640 1 -- 172.21.15.39:0/4054101158 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11917 (secure 0 0 0) 0x7ff864076a10 con 0x7ff8641094a0 2024-09-21T01:14:03.576 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:14:03.576 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"btime":"2024-09-21T01:12:25:429703+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:12:24.519635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:14:03.576 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-09-21T01:14:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8480b6680 msgr2=0x7ff8480b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8480b6680 0x7ff8480b8b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff864077160 tx=0x7ff854005f90 comp rx=0 tx=0).stop 2024-09-21T01:14:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 msgr2=0x7ff864075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 0x7ff864075c50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff86406c9f0 tx=0x7ff85000ee50 comp rx=0 tx=0).stop 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 shutdown_connections 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff864109df0 0x7ff864076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8641094a0 0x7ff864075c50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff8480b6680 0x7ff8480b8b90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 --2- 172.21.15.39:0/4054101158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff864103b60 0x7ff864075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 >> 172.21.15.39:0/4054101158 conn(0x7ff86407b850 msgr2=0x7ff864107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 shutdown_connections 2024-09-21T01:14:03.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:03.577+0000 7ff869800640 1 -- 172.21.15.39:0/4054101158 wait complete. 2024-09-21T01:14:03.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:03 smithi039 ceph-mon[30415]: pgmap v713: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 875 KiB/s wr, 35 op/s 2024-09-21T01:14:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:03 smithi150 ceph-mon[34945]: pgmap v713: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 875 KiB/s wr, 35 op/s 2024-09-21T01:14:03.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:03 smithi160 ceph-mon[34696]: pgmap v713: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 875 KiB/s wr, 35 op/s 2024-09-21T01:14:04.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 4.00 MB/sec execute 391 sec latency 22645.987 ms 2024-09-21T01:14:04.524 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id ddad0d1 --release --await' 2024-09-21T01:14:04.524 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id ddad0d1 --release --await 2024-09-21T01:14:04.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:04 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4054101158' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:04.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:04 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4054101158' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:04.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:04 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4054101158' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:04.916 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:14:05.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430038 3.99 MB/sec execute 392 sec latency 23646.080 ms 2024-09-21T01:14:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 -- 172.21.15.39:0/3860080098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810b6c0 msgr2=0x7f8a3810bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3860080098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810b6c0 0x7f8a3810bad0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009900 tx=0x7f8a2002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 -- 172.21.15.39:0/3860080098 shutdown_connections 2024-09-21T01:14:05.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3860080098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a3810c010 0x7f8a38112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3860080098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810b6c0 0x7f8a3810bad0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3860080098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a38106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 -- 172.21.15.39:0/3860080098 >> 172.21.15.39:0/3860080098 conn(0x7f8a381017d0 msgr2=0x7f8a38103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:05.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 -- 172.21.15.39:0/3860080098 shutdown_connections 2024-09-21T01:14:05.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.559+0000 7f8a3f600640 1 -- 172.21.15.39:0/3860080098 wait complete. 2024-09-21T01:14:05.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.560+0000 7f8a3f600640 1 Processor -- start 2024-09-21T01:14:05.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.560+0000 7f8a3f600640 1 -- start start 2024-09-21T01:14:05.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.560+0000 7f8a3f600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a3819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3f600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a3810b6c0 0x7f8a3819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810c010 0x7f8a381a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38115d20 con 0x7f8a3810c010 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3f600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38115ba0 con 0x7f8a3810b6c0 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3f600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38115ea0 con 0x7f8a38105d80 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810c010 0x7f8a381a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a3819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a3819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58300/0 (socket says 172.21.15.39:58300) 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3cc00640 1 -- 172.21.15.39:0/3991502478 learned_addr learned my addr 172.21.15.39:0/3991502478 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:05.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a3810b6c0 0x7f8a3819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3d600640 1 -- 172.21.15.39:0/3991502478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 msgr2=0x7f8a3819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3d600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a3819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3d600640 1 -- 172.21.15.39:0/3991502478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a3810b6c0 msgr2=0x7f8a3819cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3d600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a3810b6c0 0x7f8a3819cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.561+0000 7f8a3d600640 1 -- 172.21.15.39:0/3991502478 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a381a2640 con 0x7f8a3810c010 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a3819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a3d600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810c010 0x7f8a381a1f10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000e940 tx=0x7f8a3000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a36000640 1 -- 172.21.15.39:0/3991502478 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30004d20 con 0x7f8a3810c010 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a36000640 1 -- 172.21.15.39:0/3991502478 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a30005020 con 0x7f8a3810c010 2024-09-21T01:14:05.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a36000640 1 -- 172.21.15.39:0/3991502478 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a300116d0 con 0x7f8a3810c010 2024-09-21T01:14:05.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a381a2930 con 0x7f8a3810c010 2024-09-21T01:14:05.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.562+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a381a9090 con 0x7f8a3810c010 2024-09-21T01:14:05.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.566+0000 7f8a4624a740 1 Processor -- start 2024-09-21T01:14:05.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.566+0000 7f8a36000640 1 -- 172.21.15.39:0/3991502478 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8a30005460 con 0x7f8a3810c010 2024-09-21T01:14:05.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.566+0000 7f8a4624a740 1 -- start start 2024-09-21T01:14:05.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.566+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee7171c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.566+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 0x557fee717700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee717c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a4624a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557fee68e290 con 0x557fee34f840 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a4624a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557fee67d6c0 con 0x557fee370520 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a4624a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557fee6814d0 con 0x557fee347660 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 0x557fee717700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 0x557fee717700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58312/0 (socket says 172.21.15.39:58312) 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 -- 172.21.15.39:0/433402101 learned_addr learned my addr 172.21.15.39:0/433402101 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee7171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee717c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 -- 172.21.15.39:0/433402101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 msgr2=0x557fee7171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee7171c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 -- 172.21.15.39:0/433402101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 msgr2=0x557fee717c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee717c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.567+0000 7f8a3cc00640 1 -- 172.21.15.39:0/433402101 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fee60a460 con 0x557fee347660 2024-09-21T01:14:05.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 0x557fee717700 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8a28011630 tx=0x7f8a28011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ce312fb8c68678 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:05.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a36000640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a1c0b6700 0x7f8a1c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a2dc00640 1 -- 172.21.15.39:0/433402101 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2801e070 con 0x557fee347660 2024-09-21T01:14:05.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a36000640 1 -- 172.21.15.39:0/3991502478 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7f8a300e7b40 con 0x7f8a3810c010 2024-09-21T01:14:05.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a2dc00640 1 -- 172.21.15.39:0/433402101 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a2800fba0 con 0x557fee347660 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a2dc00640 1 -- 172.21.15.39:0/433402101 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28019ca0 con 0x557fee347660 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.568+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a1c0b6700 0x7f8a1c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 -- 172.21.15.39:0/433402101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 msgr2=0x557fee717700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 0x557fee717700 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8a28011630 tx=0x7f8a28011af0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a1c0b6700 0x7f8a1c0b8c10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8a2002a9e0 tx=0x7f8a20005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 -- 172.21.15.39:0/433402101 shutdown_connections 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee717c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee347660 0x557fee717700 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 --2- 172.21.15.39:0/433402101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee7171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 -- 172.21.15.39:0/433402101 >> 172.21.15.39:0/433402101 conn(0x557fee364910 msgr2=0x557fee7150b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 -- 172.21.15.39:0/433402101 shutdown_connections 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.569+0000 7f8a4624a740 1 -- 172.21.15.39:0/433402101 wait complete. 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 -- start start 2024-09-21T01:14:05.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 Processor -- start 2024-09-21T01:14:05.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee722d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee34f840 0x557fee723280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee72ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557fee4debb0 con 0x557fee347660 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557fee6186c0 con 0x557fee34f840 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.570+0000 7f8a4624a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557fee7259a0 con 0x557fee370520 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a4624a740 10 client.0 _ll_register_callbacks cb 0x557fee633490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee722d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee722d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51492/0 (socket says 172.21.15.39:51492) 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 -- 172.21.15.39:0/1931217804 learned_addr learned my addr 172.21.15.39:0/1931217804 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3d600640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee72ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee34f840 0x557fee723280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 -- 172.21.15.39:0/1931217804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 msgr2=0x557fee72ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee72ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 -- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee34f840 msgr2=0x557fee723280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee34f840 0x557fee723280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3cc00640 1 -- 172.21.15.39:0/1931217804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fee3dbb00 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.571+0000 7f8a3d600640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee72ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee722d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009f40 tx=0x7f8a28019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec85b7fccbe4a69 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a26000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a26000640 1 -- 172.21.15.39:0/1931217804 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28026020 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a26000640 1 -- 172.21.15.39:0/1931217804 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7f8a2800fd50 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a2800fba0 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a26000640 1 -- 172.21.15.39:0/1931217804 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8a28031050 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a4624a740 10 client.15939 fetch_fsmap learned FSMap version 34 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.572+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557fee452960 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a26000640 1 -- 172.21.15.39:0/1931217804 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a2800fba0 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a26000640 1 -- 172.21.15.39:0/1931217804 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f8a28021070 con 0x557fee347660 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a4624a740 10 client.15939 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a4624a740 10 client.15939 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a4624a740 20 client.15939 populate_metadata read hostname 'smithi039' 2024-09-21T01:14:05.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a4624a740 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee32eea0 0x557fee734bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a4624a740 4 client.15939 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:14:05.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fee6901e0 con 0x557fee32eea0 2024-09-21T01:14:05.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.573+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee32eea0 0x557fee734bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.574+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee32eea0 0x557fee734bf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16d16d55437cb446 server_cookie=9888648fe2e529a1 in_seq=0 out_seq=0 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.574+0000 7f8a26000640 10 client.15939 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a26000640 1 -- 172.21.15.39:0/1931217804 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557fee6901e0 con 0x557fee32eea0 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a26000640 10 client.15939 handle_command_reply: tid=0 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 client.15939 shutdown 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 msgr2=0x557fee722d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee722d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009f40 tx=0x7f8a28019670 comp rx=0 tx=0).stop 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 shutdown_connections 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee32eea0 0x557fee734bf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee72ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee34f840 0x557fee723280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 --2- 172.21.15.39:0/1931217804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee722d40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 >> 172.21.15.39:0/1931217804 conn(0x557fee364910 msgr2=0x557fee71a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.575+0000 7f8a26000640 0 client.15939 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.576+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 shutdown_connections 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.576+0000 7f8a4624a740 1 -- 172.21.15.39:0/1931217804 wait complete. 2024-09-21T01:14:05.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.576+0000 7f8a4624a740 20 client.15939 trim_cache size 0 max 16384 2024-09-21T01:14:05.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 Processor -- start 2024-09-21T01:14:05.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 -- start start 2024-09-21T01:14:05.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee347660 0x557fee72ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee72f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee71fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b5a80 con 0x557fee34f840 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b32f0 con 0x557fee347660 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.594+0000 7f8a4624a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b1160 con 0x557fee370520 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a2fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee72f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee347660 0x557fee72ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a2fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee72f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51496/0 (socket says 172.21.15.39:51496) 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a2fe00640 1 -- 172.21.15.39:0/3717223917 learned_addr learned my addr 172.21.15.39:0/3717223917 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3d600640 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee71fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 -- 172.21.15.39:0/3717223917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 msgr2=0x557fee71fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee71fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 -- 172.21.15.39:0/3717223917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 msgr2=0x557fee72f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee72f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 -- 172.21.15.39:0/3717223917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fee71e3d0 con 0x557fee347660 2024-09-21T01:14:05.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.595+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee347660 0x557fee72ebc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a28004a60 tx=0x7f8a2801a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9cc6830ae92710 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.596+0000 7f8a2d200640 1 -- 172.21.15.39:0/3717223917 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28026040 con 0x557fee347660 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.596+0000 7f8a2d200640 1 -- 172.21.15.39:0/3717223917 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a28004680 con 0x557fee347660 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.596+0000 7f8a2d200640 1 -- 172.21.15.39:0/3717223917 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2801e040 con 0x557fee347660 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.596+0000 7f8a4624a740 1 -- 172.21.15.39:0/3717223917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee347660 msgr2=0x557fee72ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.596+0000 7f8a4624a740 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee347660 0x557fee72ebc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a28004a60 tx=0x7f8a2801a980 comp rx=0 tx=0).stop 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 -- 172.21.15.39:0/3717223917 shutdown_connections 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee370520 0x557fee71fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee34f840 0x557fee72f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 --2- 172.21.15.39:0/3717223917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee347660 0x557fee72ebc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 -- 172.21.15.39:0/3717223917 >> 172.21.15.39:0/3717223917 conn(0x557fee364910 msgr2=0x557fee7213c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:05.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 -- 172.21.15.39:0/3717223917 shutdown_connections 2024-09-21T01:14:05.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 -- 172.21.15.39:0/3717223917 wait complete. 2024-09-21T01:14:05.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 -- start start 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.597+0000 7f8a4624a740 1 Processor -- start 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee72b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee34f840 0x557fee726920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee726ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b17b0 con 0x557fee347660 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b2c20 con 0x557fee370520 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b1e80 con 0x557fee34f840 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a4624a740 10 client.0 _ll_register_callbacks cb 0x7f8a2801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee72b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee72b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51504/0 (socket says 172.21.15.39:51504) 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a3cc00640 1 -- 172.21.15.39:0/973770380 learned_addr learned my addr 172.21.15.39:0/973770380 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a3d600640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee726ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.598+0000 7f8a2fe00640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee34f840 0x557fee726920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3d600640 1 -- 172.21.15.39:0/973770380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee34f840 msgr2=0x557fee726920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3d600640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee34f840 0x557fee726920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3d600640 1 -- 172.21.15.39:0/973770380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 msgr2=0x557fee72b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3d600640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee72b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3d600640 1 -- 172.21.15.39:0/973770380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fee727650 con 0x557fee370520 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee72b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a3d600640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee726ec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8a300e4e50 tx=0x7f8a3000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50a7eb8cbe149717 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a26000640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a26000640 1 -- 172.21.15.39:0/973770380 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3001f070 con 0x557fee370520 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a28004680 con 0x557fee370520 2024-09-21T01:14:05.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a26000640 1 -- 172.21.15.39:0/973770380 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7f8a300ac2d0 con 0x557fee370520 2024-09-21T01:14:05.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a26000640 1 -- 172.21.15.39:0/973770380 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8a300ad110 con 0x557fee370520 2024-09-21T01:14:05.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a4624a740 10 client.25973 fetch_fsmap learned FSMap version 34 2024-09-21T01:14:05.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.599+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557fee79f560 con 0x557fee370520 2024-09-21T01:14:05.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a26000640 1 -- 172.21.15.39:0/973770380 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a28004680 con 0x557fee370520 2024-09-21T01:14:05.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a26000640 1 -- 172.21.15.39:0/973770380 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f8a3000f040 con 0x557fee370520 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a4624a740 10 client.25973 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a4624a740 10 client.25973 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a4624a740 20 client.25973 populate_metadata read hostname 'smithi039' 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a4624a740 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee304250 0x557fee7a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a4624a740 4 client.25973 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ddad0d1", "release": true, "await": true}] 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.600+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ddad0d1", "release": true, "await": true}) -- 0x557fee68bc90 con 0x557fee304250 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.601+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee304250 0x557fee7a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.601+0000 7f8a3cc00640 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee304250 0x557fee7a1d80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc0f7dd609e25413 server_cookie=97e2076c1124d08e in_seq=0 out_seq=0 2024-09-21T01:14:05.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.601+0000 7f8a26000640 10 client.25973 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:05.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a26000640 1 -- 172.21.15.39:0/973770380 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557fee68bc90 con 0x557fee304250 2024-09-21T01:14:05.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a26000640 10 client.25973 handle_command_reply: tid=0 2024-09-21T01:14:05.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 client.25973 shutdown 2024-09-21T01:14:05.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 msgr2=0x557fee726ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee726ec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8a300e4e50 tx=0x7f8a3000cef0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 shutdown_connections 2024-09-21T01:14:05.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557fee304250 0x557fee7a1d80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557fee370520 0x557fee726ec0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557fee34f840 0x557fee726920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 --2- 172.21.15.39:0/973770380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557fee347660 0x557fee72b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 >> 172.21.15.39:0/973770380 conn(0x557fee364910 msgr2=0x557fee7181f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:05.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.607+0000 7f8a26000640 0 client.25973 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:05.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.608+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 shutdown_connections 2024-09-21T01:14:05.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.608+0000 7f8a4624a740 1 -- 172.21.15.39:0/973770380 wait complete. 2024-09-21T01:14:05.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.608+0000 7f8a4624a740 20 client.25973 trim_cache size 0 max 16384 2024-09-21T01:14:05.610 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 34, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 40, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "ddad0d1": { 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 40, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:14:05.611 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 144.5, 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:05.612 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a1c0b6700 msgr2=0x7f8a1c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a1c0b6700 0x7f8a1c0b8c10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8a2002a9e0 tx=0x7f8a20005a20 comp rx=0 tx=0).stop 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810c010 msgr2=0x7f8a381a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810c010 0x7f8a381a1f10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000e940 tx=0x7f8a3000ee00 comp rx=0 tx=0).stop 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 shutdown_connections 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a3810c010 0x7f8a381a1f10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a1c0b6700 0x7f8a1c0b8c10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a3810b6c0 0x7f8a3819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 --2- 172.21.15.39:0/3991502478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a38105d80 0x7f8a3819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:05.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 >> 172.21.15.39:0/3991502478 conn(0x7f8a381017d0 msgr2=0x7f8a38103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:05.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 shutdown_connections 2024-09-21T01:14:05.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:05.609+0000 7f8a3f600640 1 -- 172.21.15.39:0/3991502478 wait complete. 2024-09-21T01:14:05.632 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:05 smithi039 ceph-mon[30415]: pgmap v714: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 886 KiB/s wr, 51 op/s 2024-09-21T01:14:05.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:05 smithi150 ceph-mon[34945]: pgmap v714: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 886 KiB/s wr, 51 op/s 2024-09-21T01:14:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:05 smithi160 ceph-mon[34696]: pgmap v714: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 886 KiB/s wr, 51 op/s 2024-09-21T01:14:06.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430648 3.99 MB/sec execute 393 sec latency 24066.874 ms 2024-09-21T01:14:06.670 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:06 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:14:06.672 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ddad0d1', total seconds elapsed: 29.0 2024-09-21T01:14:06.672 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-21T01:14:06.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:14:06.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:06 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:14:07.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:06 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:14:07.124 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:14:07.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430914 3.98 MB/sec execute 394 sec latency 727.513 ms 2024-09-21T01:14:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:07 smithi039 ceph-mon[30415]: pgmap v715: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 734 KiB/s wr, 48 op/s 2024-09-21T01:14:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:14:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:14:07.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:14:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:07 smithi160 ceph-mon[34696]: pgmap v715: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 734 KiB/s wr, 48 op/s 2024-09-21T01:14:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:14:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:14:07.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:14:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.886+0000 7ff1bd800640 1 -- 172.21.15.39:0/66301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b81094a0 msgr2=0x7ff1b81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.886+0000 7ff1bd800640 1 --2- 172.21.15.39:0/66301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b81094a0 0x7ff1b81098b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009900 tx=0x7ff1a002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:14:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.886+0000 7ff1bd800640 1 -- 172.21.15.39:0/66301606 shutdown_connections 2024-09-21T01:14:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.886+0000 7ff1bd800640 1 --2- 172.21.15.39:0/66301606 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1b8109df0 0x7ff1b8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.887+0000 7ff1bd800640 1 --2- 172.21.15.39:0/66301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b81094a0 0x7ff1b81098b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.887+0000 7ff1bd800640 1 --2- 172.21.15.39:0/66301606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1b8103b60 0x7ff1b8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.887+0000 7ff1bd800640 1 -- 172.21.15.39:0/66301606 >> 172.21.15.39:0/66301606 conn(0x7ff1b807b850 msgr2=0x7ff1b807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.887+0000 7ff1bd800640 1 -- 172.21.15.39:0/66301606 shutdown_connections 2024-09-21T01:14:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.887+0000 7ff1bd800640 1 -- 172.21.15.39:0/66301606 wait complete. 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 Processor -- start 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 -- start start 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 0x7ff1b819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1b81094a0 0x7ff1b819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1b8109df0 0x7ff1b81a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8113ab0 con 0x7ff1b81094a0 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8113930 con 0x7ff1b8103b60 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.888+0000 7ff1bd800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8113c30 con 0x7ff1b8109df0 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 0x7ff1b819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 0x7ff1b819c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50410/0 (socket says 172.21.15.39:50410) 2024-09-21T01:14:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 -- 172.21.15.39:0/3978143682 learned_addr learned my addr 172.21.15.39:0/3978143682 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:07.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1afe00640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1b81094a0 0x7ff1b819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:07.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b7400640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1b8109df0 0x7ff1b81a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:07.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 -- 172.21.15.39:0/3978143682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1b8109df0 msgr2=0x7ff1b81a1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1b8109df0 0x7ff1b81a1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 -- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1b81094a0 msgr2=0x7ff1b819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1b81094a0 0x7ff1b819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.889+0000 7ff1b6a00640 1 -- 172.21.15.39:0/3978143682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b81a2690 con 0x7ff1b8103b60 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.890+0000 7ff1b6a00640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 0x7ff1b819c720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800e9f0 tx=0x7ff1a800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.890+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a800ccc0 con 0x7ff1b8103b60 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.890+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a80108e0 con 0x7ff1b8103b60 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.890+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8018880 con 0x7ff1b8103b60 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.890+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b81a2980 con 0x7ff1b8103b60 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.890+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b81a90e0 con 0x7ff1b8103b60 2024-09-21T01:14:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.891+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b8103f70 con 0x7ff1b8103b60 2024-09-21T01:14:07.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.893+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff1a8010430 con 0x7ff1b8103b60 2024-09-21T01:14:07.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.895+0000 7ff1af400640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff19c0b6700 0x7ff19c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:07.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.895+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9511+0+0 (secure 0 0 0) 0x7ff1a80e8390 con 0x7ff1b8103b60 2024-09-21T01:14:07.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.895+0000 7ff1afe00640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff19c0b6700 0x7ff19c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:07.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.896+0000 7ff1afe00640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff19c0b6700 0x7ff19c0b8c10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009230 tx=0x7ff1a0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:07.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:07.896+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff1a80ae4a0 con 0x7ff1b8103b60 2024-09-21T01:14:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:07 smithi150 ceph-mon[34945]: pgmap v715: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 734 KiB/s wr, 48 op/s 2024-09-21T01:14:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:14:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:14:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:14:08.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.133+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1b81a9410 con 0x7ff1b8103b60 2024-09-21T01:14:08.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.134+0000 7ff1af400640 1 -- 172.21.15.39:0/3978143682 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11917 (secure 0 0 0) 0x7ff1a80b2880 con 0x7ff1b8103b60 2024-09-21T01:14:08.135 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:14:08.135 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"btime":"2024-09-21T01:12:25:429703+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:12:24.519635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:14:08.135 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.136+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff19c0b6700 msgr2=0x7ff19c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.136+0000 7ff1bd800640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff19c0b6700 0x7ff19c0b8c10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009230 tx=0x7ff1a0031400 comp rx=0 tx=0).stop 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.136+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 msgr2=0x7ff1b819c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.136+0000 7ff1bd800640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 0x7ff1b819c720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800e9f0 tx=0x7ff1a800edf0 comp rx=0 tx=0).stop 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 shutdown_connections 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1b8109df0 0x7ff1b81a1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1b81094a0 0x7ff1b819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff19c0b6700 0x7ff19c0b8c10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:08.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 --2- 172.21.15.39:0/3978143682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b8103b60 0x7ff1b819c720 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:08.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 >> 172.21.15.39:0/3978143682 conn(0x7ff1b807b850 msgr2=0x7ff1b8107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:08.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 shutdown_connections 2024-09-21T01:14:08.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:08.137+0000 7ff1bd800640 1 -- 172.21.15.39:0/3978143682 wait complete. 2024-09-21T01:14:08.186 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430914 3.97 MB/sec execute 395 sec latency 1138.999 ms 2024-09-21T01:14:08.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:08 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3978143682' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:08.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:08 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3978143682' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:09.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:08 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3978143682' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:09.072 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-21T01:14:09.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-21T01:14:09.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 430914 3.96 MB/sec execute 396 sec latency 2139.057 ms 2024-09-21T01:14:09.463 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:14:09.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:09 smithi160 ceph-mon[34696]: pgmap v716: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.0 MiB/s wr, 67 op/s 2024-09-21T01:14:09.868 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:09 smithi039 ceph-mon[30415]: pgmap v716: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.0 MiB/s wr, 67 op/s 2024-09-21T01:14:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:09 smithi150 ceph-mon[34945]: pgmap v716: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 1.0 MiB/s wr, 67 op/s 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 -- 172.21.15.39:0/3885193927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c1039c0 msgr2=0x7fac0c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 --2- 172.21.15.39:0/3885193927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c1039c0 0x7fac0c103dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009900 tx=0x7fabfc02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 -- 172.21.15.39:0/3885193927 shutdown_connections 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 --2- 172.21.15.39:0/3885193927 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c109c50 0x7fac0c110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 --2- 172.21.15.39:0/3885193927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac0c109300 0x7fac0c109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 --2- 172.21.15.39:0/3885193927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c1039c0 0x7fac0c103dd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 -- 172.21.15.39:0/3885193927 >> 172.21.15.39:0/3885193927 conn(0x7fac0c0ff430 msgr2=0x7fac0c101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:10.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 -- 172.21.15.39:0/3885193927 shutdown_connections 2024-09-21T01:14:10.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.178+0000 7fac10e00640 1 -- 172.21.15.39:0/3885193927 wait complete. 2024-09-21T01:14:10.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.179+0000 7fac10e00640 1 Processor -- start 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 -- start start 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 0x7fac0c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac0c109300 0x7fac0c19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c109c50 0x7fac0c19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c113910 con 0x7fac0c109300 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c113790 con 0x7fac0c109c50 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac10e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c113a90 con 0x7fac0c1039c0 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac0b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 0x7fac0c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac0b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 0x7fac0c19a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58372/0 (socket says 172.21.15.39:58372) 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.180+0000 7fac0b400640 1 -- 172.21.15.39:0/2365933488 learned_addr learned my addr 172.21.15.39:0/2365933488 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0aa00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac0c109300 0x7fac0c19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0be00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c109c50 0x7fac0c19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0b400640 1 -- 172.21.15.39:0/2365933488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c109c50 msgr2=0x7fac0c19fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0b400640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c109c50 0x7fac0c19fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0b400640 1 -- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac0c109300 msgr2=0x7fac0c19a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0b400640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac0c109300 0x7fac0c19a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0b400640 1 -- 172.21.15.39:0/2365933488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0c1a0270 con 0x7fac0c1039c0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0be00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c109c50 0x7fac0c19fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac0b400640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 0x7fac0c19a300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fabfc005940 tx=0x7fabfc0040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac03e00640 1 -- 172.21.15.39:0/2365933488 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc03f030 con 0x7fac0c1039c0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac03e00640 1 -- 172.21.15.39:0/2365933488 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabfc0044c0 con 0x7fac0c1039c0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.181+0000 7fac03e00640 1 -- 172.21.15.39:0/2365933488 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc035070 con 0x7fac0c1039c0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.182+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0c1a0500 con 0x7fac0c1039c0 2024-09-21T01:14:10.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.182+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac0c1a6cc0 con 0x7fac0c1039c0 2024-09-21T01:14:10.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.185+0000 7fac19039740 1 Processor -- start 2024-09-21T01:14:10.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432748 3.98 MB/sec execute 397 sec latency 2333.132 ms 2024-09-21T01:14:10.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.185+0000 7fac19039740 1 -- start start 2024-09-21T01:14:10.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac19039740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 0x5626a2514920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac19039740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2514e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac19039740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2147b60 0x5626a25153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac19039740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5626a2485ca0 con 0x5626a2147b60 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac19039740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5626a24750d0 con 0x5626a213f980 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac19039740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5626a2478ee0 con 0x5626a2168840 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac0be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 0x5626a2514920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac0be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 0x5626a2514920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58380/0 (socket says 172.21.15.39:58380) 2024-09-21T01:14:10.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.186+0000 7fac0be00640 1 -- 172.21.15.39:0/1280780146 learned_addr learned my addr 172.21.15.39:0/1280780146 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0aa00640 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2147b60 0x5626a25153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0be00640 1 -- 172.21.15.39:0/1280780146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 msgr2=0x5626a2514e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:14:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0be00640 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2514e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0be00640 1 -- 172.21.15.39:0/1280780146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2147b60 msgr2=0x5626a25153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0be00640 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2147b60 0x5626a25153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0be00640 1 -- 172.21.15.39:0/1280780146 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626a2427400 con 0x5626a2168840 2024-09-21T01:14:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac03e00640 1 -- 172.21.15.39:0/2365933488 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fabfc042030 con 0x7fac0c1039c0 2024-09-21T01:14:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.187+0000 7fac0be00640 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 0x5626a2514920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fac04009fd0 tx=0x7fac040108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83871e05296d8ee7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.188+0000 7fac00c00640 1 -- 172.21.15.39:0/1280780146 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04011760 con 0x5626a2168840 2024-09-21T01:14:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.188+0000 7fac00c00640 1 -- 172.21.15.39:0/1280780146 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac04011db0 con 0x5626a2168840 2024-09-21T01:14:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.188+0000 7fac00c00640 1 -- 172.21.15.39:0/1280780146 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0400e3c0 con 0x5626a2168840 2024-09-21T01:14:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 -- 172.21.15.39:0/1280780146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 msgr2=0x5626a2514920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 0x5626a2514920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fac04009fd0 tx=0x7fac040108e0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 -- 172.21.15.39:0/1280780146 shutdown_connections 2024-09-21T01:14:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2147b60 0x5626a25153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2514e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 --2- 172.21.15.39:0/1280780146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2168840 0x5626a2514920 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 -- 172.21.15.39:0/1280780146 >> 172.21.15.39:0/1280780146 conn(0x5626a215c990 msgr2=0x5626a2512810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac03e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fabf00b6670 0x7fabf00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 -- 172.21.15.39:0/1280780146 shutdown_connections 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac03e00640 1 -- 172.21.15.39:0/2365933488 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fabfc107ce0 con 0x7fac0c1039c0 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac19039740 1 -- 172.21.15.39:0/1280780146 wait complete. 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.189+0000 7fac0be00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fabf00b6670 0x7fabf00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.190+0000 7fac19039740 1 -- start start 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.190+0000 7fac0be00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fabf00b6670 0x7fabf00b8b80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fac04005bf0 tx=0x7fac04010f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.190+0000 7fac19039740 1 Processor -- start 2024-09-21T01:14:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2520030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2520570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a252c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5626a22d67a0 con 0x5626a213f980 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5626a241aa10 con 0x5626a2168840 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5626a2523150 con 0x5626a2147b60 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac19039740 10 client.0 _ll_register_callbacks cb 0x5626a242abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2520570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2520570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58390/0 (socket says 172.21.15.39:58390) 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac0aa00640 1 -- 172.21.15.39:0/2841718901 learned_addr learned my addr 172.21.15.39:0/2841718901 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac0b400640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2520030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.191+0000 7fac0be00640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a252c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac0aa00640 1 -- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 msgr2=0x5626a252c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac0aa00640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a252c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac0aa00640 1 -- 172.21.15.39:0/2841718901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 msgr2=0x5626a2520030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac0aa00640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2520030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac0aa00640 1 -- 172.21.15.39:0/2841718901 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626a21e2b90 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac0aa00640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2520570 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fabf8012480 tx=0x7fabf8012940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd227ffd137b2b19 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fabf5600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fabf5600640 1 -- 172.21.15.39:0/2841718901 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8004020 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fabf5600640 1 -- 172.21.15.39:0/2841718901 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fabf80041c0 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.192+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac04011db0 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fabf5600640 1 -- 172.21.15.39:0/2841718901 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fabf8025050 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 10 client.25981 fetch_fsmap learned FSMap version 35 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626a252ef20 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fabf5600640 1 -- 172.21.15.39:0/2841718901 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac04011db0 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fabf5600640 1 -- 172.21.15.39:0/2841718901 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fabf8003890 con 0x5626a2147b60 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 10 client.25981 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T01:14:10.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 10 client.25981 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:14:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 20 client.25981 populate_metadata read hostname 'smithi039' 2024-09-21T01:14:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a21271c0 0x5626a2532440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 4 client.25981 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:14:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.193+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626a2487bf0 con 0x5626a21271c0 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.194+0000 7fac0b400640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a21271c0 0x5626a2532440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.195+0000 7fac0b400640 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a21271c0 0x5626a2532440 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5912d972e4ba1635 server_cookie=4f5b0d5688267680 in_seq=0 out_seq=0 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.195+0000 7fabf5600640 10 client.25981 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.195+0000 7fabf5600640 1 -- 172.21.15.39:0/2841718901 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5626a2487bf0 con 0x5626a21271c0 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.195+0000 7fabf5600640 10 client.25981 handle_command_reply: tid=0 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.195+0000 7fac19039740 1 client.25981 shutdown 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 msgr2=0x5626a2520570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2520570 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fabf8012480 tx=0x7fabf8012940 comp rx=0 tx=0).stop 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 shutdown_connections 2024-09-21T01:14:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a21271c0 0x5626a2532440 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a252c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2520570 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 --2- 172.21.15.39:0/2841718901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2520030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fabf5600640 0 client.25981 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 >> 172.21.15.39:0/2841718901 conn(0x5626a215c990 msgr2=0x5626a2517ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 shutdown_connections 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 1 -- 172.21.15.39:0/2841718901 wait complete. 2024-09-21T01:14:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.196+0000 7fac19039740 20 client.25981 trim_cache size 0 max 16384 2024-09-21T01:14:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.216+0000 7fac19039740 1 Processor -- start 2024-09-21T01:14:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 -- start start 2024-09-21T01:14:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2521f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a252c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a251d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1b1690 con 0x5626a213f980 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1aef00 con 0x5626a2168840 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac19039740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1acd70 con 0x5626a2147b60 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a252c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac0be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a251d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac0b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2521f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a252c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58392/0 (socket says 172.21.15.39:58392) 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac0be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a251d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50444/0 (socket says 172.21.15.39:50444) 2024-09-21T01:14:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.217+0000 7fac0aa00640 1 -- 172.21.15.39:0/952926321 learned_addr learned my addr 172.21.15.39:0/952926321 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0aa00640 1 -- 172.21.15.39:0/952926321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 msgr2=0x5626a251d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0aa00640 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a251d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0aa00640 1 -- 172.21.15.39:0/952926321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 msgr2=0x5626a2521f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0aa00640 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2521f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0aa00640 1 -- 172.21.15.39:0/952926321 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626a25212f0 con 0x5626a2147b60 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0be00640 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a251d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fac0aa00640 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a252c340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fabf8003d30 tx=0x7fabf801a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a70868839602053 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fabf7e00640 1 -- 172.21.15.39:0/952926321 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf80038f0 con 0x5626a2147b60 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fabf7e00640 1 -- 172.21.15.39:0/952926321 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabf8003a90 con 0x5626a2147b60 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.218+0000 7fabf7e00640 1 -- 172.21.15.39:0/952926321 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8005030 con 0x5626a2147b60 2024-09-21T01:14:10.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 -- 172.21.15.39:0/952926321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 msgr2=0x5626a252c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a252c340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fabf8003d30 tx=0x7fabf801a620 comp rx=0 tx=0).stop 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 -- 172.21.15.39:0/952926321 shutdown_connections 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a2168840 0x5626a251d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a252c340 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 --2- 172.21.15.39:0/952926321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a213f980 0x5626a2521f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.219+0000 7fac19039740 1 -- 172.21.15.39:0/952926321 >> 172.21.15.39:0/952926321 conn(0x5626a215c990 msgr2=0x5626a25126f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.220+0000 7fac19039740 1 -- 172.21.15.39:0/952926321 shutdown_connections 2024-09-21T01:14:10.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.220+0000 7fac19039740 1 -- 172.21.15.39:0/952926321 wait complete. 2024-09-21T01:14:10.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.220+0000 7fac19039740 1 -- start start 2024-09-21T01:14:10.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.220+0000 7fac19039740 1 Processor -- start 2024-09-21T01:14:10.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2524540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2524a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2168840 0x5626a2524fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1ad3c0 con 0x5626a2168840 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1ae830 con 0x5626a213f980 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1ada90 con 0x5626a2147b60 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac19039740 10 client.0 _ll_register_callbacks cb 0x7fabf8009450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2524a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac0b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2524540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2524a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:58396/0 (socket says 172.21.15.39:58396) 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac0aa00640 1 -- 172.21.15.39:0/2158398439 learned_addr learned my addr 172.21.15.39:0/2158398439 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.221+0000 7fac0be00640 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2168840 0x5626a2524fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac0b400640 1 -- 172.21.15.39:0/2158398439 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 msgr2=0x5626a2524a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac0b400640 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2524a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac0b400640 1 -- 172.21.15.39:0/2158398439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2168840 msgr2=0x5626a2524fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:10.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac0b400640 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2168840 0x5626a2524fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac0b400640 1 -- 172.21.15.39:0/2158398439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626a259c040 con 0x5626a213f980 2024-09-21T01:14:10.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac0b400640 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2524540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabfc104fa0 tx=0x7fabfc0c9700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6c4960ccdca574 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:10.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fabf5600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:14:10.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fabf5600640 1 -- 172.21.15.39:0/2158398439 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc049040 con 0x5626a213f980 2024-09-21T01:14:10.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fabf5600640 1 -- 172.21.15.39:0/2158398439 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fabfc0ccb00 con 0x5626a213f980 2024-09-21T01:14:10.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabf8003a90 con 0x5626a213f980 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fabf5600640 1 -- 172.21.15.39:0/2158398439 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fabfc0d2180 con 0x5626a213f980 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac19039740 10 client.25994 fetch_fsmap learned FSMap version 35 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.222+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626a259d260 con 0x5626a213f980 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fabf5600640 1 -- 172.21.15.39:0/2158398439 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabf8003a90 con 0x5626a213f980 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fabf5600640 1 -- 172.21.15.39:0/2158398439 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fabfc04c060 con 0x5626a213f980 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac19039740 10 client.25994 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T01:14:10.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac19039740 10 client.25994 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac19039740 20 client.25994 populate_metadata read hostname 'smithi039' 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac19039740 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a20fc570 0x5626a259fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac19039740 4 client.25994 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x5626a24836a0 con 0x5626a20fc570 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.223+0000 7fac0aa00640 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a20fc570 0x5626a259fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.224+0000 7fac0aa00640 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a20fc570 0x5626a259fa70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b87827accf20d0a server_cookie=675f6f5a3792f04b in_seq=0 out_seq=0 2024-09-21T01:14:10.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.224+0000 7fabf5600640 10 client.25994 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:10.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.399+0000 7fabf5600640 1 -- 172.21.15.39:0/2158398439 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5626a24836a0 con 0x5626a20fc570 2024-09-21T01:14:10.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.399+0000 7fabf5600640 10 client.25994 handle_command_reply: tid=0 2024-09-21T01:14:10.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.399+0000 7fac19039740 1 client.25994 shutdown 2024-09-21T01:14:10.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 msgr2=0x5626a2524540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2524540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabfc104fa0 tx=0x7fabfc0c9700 comp rx=0 tx=0).stop 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 shutdown_connections 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5626a20fc570 0x5626a259fa70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626a2168840 0x5626a2524fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5626a2147b60 0x5626a2524a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 --2- 172.21.15.39:0/2158398439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5626a213f980 0x5626a2524540 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fabf5600640 0 client.25994 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 >> 172.21.15.39:0/2158398439 conn(0x5626a215c990 msgr2=0x5626a2515950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 shutdown_connections 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 1 -- 172.21.15.39:0/2158398439 wait complete. 2024-09-21T01:14:10.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.400+0000 7fac19039740 20 client.25994 trim_cache size 0 max 16384 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 35, 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 42, 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "97fbf2fb": { 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 42, 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:14:10.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 144.5, 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:10.404 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:14:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fabf00b6670 msgr2=0x7fabf00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fabf00b6670 0x7fabf00b8b80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fac04005bf0 tx=0x7fac04010f90 comp rx=0 tx=0).stop 2024-09-21T01:14:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 msgr2=0x7fac0c19a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 0x7fac0c19a300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fabfc005940 tx=0x7fabfc0040c0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 shutdown_connections 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fabf00b6670 0x7fabf00b8b80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac0c109c50 0x7fac0c19fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac0c109300 0x7fac0c19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 --2- 172.21.15.39:0/2365933488 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fac0c1039c0 0x7fac0c19a300 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 >> 172.21.15.39:0/2365933488 conn(0x7fac0c0ff430 msgr2=0x7fac0c101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 shutdown_connections 2024-09-21T01:14:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:10.401+0000 7fac10e00640 1 -- 172.21.15.39:0/2365933488 wait complete. 2024-09-21T01:14:10.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:10 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:10.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:10 smithi039 ceph-mon[30415]: osdmap e99: 12 total, 12 up, 12 in 2024-09-21T01:14:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:10 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:10 smithi150 ceph-mon[34945]: osdmap e99: 12 total, 12 up, 12 in 2024-09-21T01:14:11.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:10 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:11.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:10 smithi160 ceph-mon[34696]: osdmap e99: 12 total, 12 up, 12 in 2024-09-21T01:14:11.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.97 MB/sec execute 398 sec latency 875.054 ms 2024-09-21T01:14:11.220 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '97fbf2fb', quiesce took 4.5 seconds. Will release after: 20.0 2024-09-21T01:14:11.693 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:11 smithi039 ceph-mon[30415]: pgmap v718: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 74 op/s 2024-09-21T01:14:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:11 smithi150 ceph-mon[34945]: pgmap v718: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 74 op/s 2024-09-21T01:14:12.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:11 smithi160 ceph-mon[34696]: pgmap v718: 129 pgs: 129 active+clean; 304 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 74 op/s 2024-09-21T01:14:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.96 MB/sec execute 399 sec latency 1875.131 ms 2024-09-21T01:14:13.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.95 MB/sec execute 400 sec latency 2875.219 ms 2024-09-21T01:14:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:13 smithi039 ceph-mon[30415]: pgmap v719: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 904 KiB/s wr, 76 op/s 2024-09-21T01:14:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:13 smithi150 ceph-mon[34945]: pgmap v719: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 904 KiB/s wr, 76 op/s 2024-09-21T01:14:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:13 smithi160 ceph-mon[34696]: pgmap v719: 129 pgs: 129 active+clean; 303 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 904 KiB/s wr, 76 op/s 2024-09-21T01:14:14.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.94 MB/sec execute 401 sec latency 3875.307 ms 2024-09-21T01:14:15.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.93 MB/sec execute 402 sec latency 4875.454 ms 2024-09-21T01:14:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:15 smithi039 ceph-mon[30415]: pgmap v720: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 968 KiB/s wr, 62 op/s 2024-09-21T01:14:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:15 smithi150 ceph-mon[34945]: pgmap v720: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 968 KiB/s wr, 62 op/s 2024-09-21T01:14:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:15 smithi160 ceph-mon[34696]: pgmap v720: 129 pgs: 129 active+clean; 305 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 968 KiB/s wr, 62 op/s 2024-09-21T01:14:16.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.92 MB/sec execute 403 sec latency 5875.545 ms 2024-09-21T01:14:17.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.91 MB/sec execute 404 sec latency 6875.646 ms 2024-09-21T01:14:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:17 smithi039 ceph-mon[30415]: pgmap v721: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-09-21T01:14:17.909 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:17 smithi160 ceph-mon[34696]: pgmap v721: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-09-21T01:14:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:17 smithi150 ceph-mon[34945]: pgmap v721: 129 pgs: 129 active+clean; 306 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-09-21T01:14:18.187 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.90 MB/sec execute 405 sec latency 7875.755 ms 2024-09-21T01:14:19.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.89 MB/sec execute 406 sec latency 8875.952 ms 2024-09-21T01:14:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:19 smithi039 ceph-mon[30415]: pgmap v722: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 625 KiB/s wr, 55 op/s 2024-09-21T01:14:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:19 smithi039 ceph-mon[30415]: osdmap e100: 12 total, 12 up, 12 in 2024-09-21T01:14:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:19 smithi150 ceph-mon[34945]: pgmap v722: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 625 KiB/s wr, 55 op/s 2024-09-21T01:14:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:19 smithi150 ceph-mon[34945]: osdmap e100: 12 total, 12 up, 12 in 2024-09-21T01:14:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:19 smithi160 ceph-mon[34696]: pgmap v722: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 625 KiB/s wr, 55 op/s 2024-09-21T01:14:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:19 smithi160 ceph-mon[34696]: osdmap e100: 12 total, 12 up, 12 in 2024-09-21T01:14:20.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.88 MB/sec execute 407 sec latency 9876.031 ms 2024-09-21T01:14:21.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.87 MB/sec execute 408 sec latency 10876.105 ms 2024-09-21T01:14:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:21 smithi039 ceph-mon[30415]: pgmap v724: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 544 KiB/s wr, 49 op/s 2024-09-21T01:14:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:21 smithi150 ceph-mon[34945]: pgmap v724: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 544 KiB/s wr, 49 op/s 2024-09-21T01:14:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:21 smithi160 ceph-mon[34696]: pgmap v724: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 544 KiB/s wr, 49 op/s 2024-09-21T01:14:22.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.86 MB/sec execute 409 sec latency 11876.216 ms 2024-09-21T01:14:23.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.85 MB/sec execute 410 sec latency 12876.330 ms 2024-09-21T01:14:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:23 smithi039 ceph-mon[30415]: pgmap v725: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 465 KiB/s wr, 39 op/s 2024-09-21T01:14:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:23 smithi150 ceph-mon[34945]: pgmap v725: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 465 KiB/s wr, 39 op/s 2024-09-21T01:14:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:23 smithi160 ceph-mon[34696]: pgmap v725: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 465 KiB/s wr, 39 op/s 2024-09-21T01:14:24.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.84 MB/sec execute 411 sec latency 13876.391 ms 2024-09-21T01:14:25.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.83 MB/sec execute 412 sec latency 14876.462 ms 2024-09-21T01:14:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:25 smithi039 ceph-mon[30415]: pgmap v726: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 28 op/s 2024-09-21T01:14:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:25 smithi150 ceph-mon[34945]: pgmap v726: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 28 op/s 2024-09-21T01:14:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:25 smithi160 ceph-mon[34696]: pgmap v726: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 28 op/s 2024-09-21T01:14:26.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.83 MB/sec execute 413 sec latency 15876.557 ms 2024-09-21T01:14:27.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.82 MB/sec execute 414 sec latency 16876.604 ms 2024-09-21T01:14:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:27 smithi039 ceph-mon[30415]: pgmap v727: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 62 KiB/s wr, 17 op/s 2024-09-21T01:14:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:27 smithi150 ceph-mon[34945]: pgmap v727: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 62 KiB/s wr, 17 op/s 2024-09-21T01:14:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:27 smithi160 ceph-mon[34696]: pgmap v727: 129 pgs: 129 active+clean; 303 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 62 KiB/s wr, 17 op/s 2024-09-21T01:14:28.188 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.81 MB/sec execute 415 sec latency 17876.650 ms 2024-09-21T01:14:29.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.80 MB/sec execute 416 sec latency 18876.719 ms 2024-09-21T01:14:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:29 smithi039 ceph-mon[30415]: pgmap v728: 129 pgs: 129 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 257 KiB/s wr, 17 op/s 2024-09-21T01:14:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:29 smithi150 ceph-mon[34945]: pgmap v728: 129 pgs: 129 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 257 KiB/s wr, 17 op/s 2024-09-21T01:14:30.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:29 smithi160 ceph-mon[34696]: pgmap v728: 129 pgs: 129 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 257 KiB/s wr, 17 op/s 2024-09-21T01:14:30.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.79 MB/sec execute 417 sec latency 19876.819 ms 2024-09-21T01:14:31.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.78 MB/sec execute 418 sec latency 20876.890 ms 2024-09-21T01:14:31.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:14:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:14:31.926 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:31 smithi039 ceph-mon[30415]: pgmap v729: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 381 KiB/s wr, 20 op/s 2024-09-21T01:14:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:31 smithi150 ceph-mon[34945]: pgmap v729: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 381 KiB/s wr, 20 op/s 2024-09-21T01:14:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:31 smithi160 ceph-mon[34696]: pgmap v729: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 381 KiB/s wr, 20 op/s 2024-09-21T01:14:32.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.77 MB/sec execute 419 sec latency 21876.989 ms 2024-09-21T01:14:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 -- 172.21.15.39:0/1802353648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8114260 msgr2=0x7fded8116730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 --2- 172.21.15.39:0/1802353648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8114260 0x7fded8116730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fded0009ca0 tx=0x7fded002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 -- 172.21.15.39:0/1802353648 shutdown_connections 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 --2- 172.21.15.39:0/1802353648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8114260 0x7fded8116730 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 --2- 172.21.15.39:0/1802353648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fded8075dd0 0x7fded80761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 --2- 172.21.15.39:0/1802353648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fded8077380 0x7fded80757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 -- 172.21.15.39:0/1802353648 >> 172.21.15.39:0/1802353648 conn(0x7fded80ff440 msgr2=0x7fded8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 -- 172.21.15.39:0/1802353648 shutdown_connections 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.234+0000 7fded7e00640 1 -- 172.21.15.39:0/1802353648 wait complete. 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.235+0000 7fded7e00640 1 Processor -- start 2024-09-21T01:14:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 -- start start 2024-09-21T01:14:32.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fded8075dd0 0x7fded81a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:32.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 0x7fded81a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:32.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fded8114260 0x7fded81a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fded8119ef0 con 0x7fded8077380 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fded8119d70 con 0x7fded8114260 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fded811a070 con 0x7fded8075dd0 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 0x7fded81a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 0x7fded81a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36844/0 (socket says 172.21.15.39:36844) 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded6000640 1 -- 172.21.15.39:0/2948068941 learned_addr learned my addr 172.21.15.39:0/2948068941 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded6a00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fded8075dd0 0x7fded81a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:32.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.236+0000 7fded7400640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fded8114260 0x7fded81a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:32.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fded6000640 1 -- 172.21.15.39:0/2948068941 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fded8075dd0 msgr2=0x7fded81a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:32.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fded6000640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fded8075dd0 0x7fded81a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fded6000640 1 -- 172.21.15.39:0/2948068941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fded8114260 msgr2=0x7fded81a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:32.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fded6000640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fded8114260 0x7fded81a60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fded6000640 1 -- 172.21.15.39:0/2948068941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded81a67d0 con 0x7fded8077380 2024-09-21T01:14:32.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fded6000640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 0x7fded81a0e00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdec400e9c0 tx=0x7fdec400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:32.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.237+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec400cd80 con 0x7fded8077380 2024-09-21T01:14:32.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.238+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdec40054d0 con 0x7fded8077380 2024-09-21T01:14:32.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.238+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4010430 con 0x7fded8077380 2024-09-21T01:14:32.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.238+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded81a6ac0 con 0x7fded8077380 2024-09-21T01:14:32.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.238+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded81ad220 con 0x7fded8077380 2024-09-21T01:14:32.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.239+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded80761e0 con 0x7fded8077380 2024-09-21T01:14:32.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.242+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fdec4004fb0 con 0x7fded8077380 2024-09-21T01:14:32.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.243+0000 7fdecf400640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdebc0b6680 0x7fdebc0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.244+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9511+0+0 (secure 0 0 0) 0x7fdec40e8170 con 0x7fded8077380 2024-09-21T01:14:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.244+0000 7fded6a00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdebc0b6680 0x7fdebc0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:32.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.245+0000 7fded6a00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdebc0b6680 0x7fdebc0b8b90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009bd0 tx=0x7fdec8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:32.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.245+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdec4056710 con 0x7fded8077380 2024-09-21T01:14:32.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.484+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fded81a1990 con 0x7fded8077380 2024-09-21T01:14:32.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.485+0000 7fdecf400640 1 -- 172.21.15.39:0/2948068941 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11915 (secure 0 0 0) 0x7fdec40ae280 con 0x7fded8077380 2024-09-21T01:14:32.486 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:14:32.486 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":35,"btime":"2024-09-21T01:14:09:598447+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:14:09.522750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:14:32.486 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 35 2024-09-21T01:14:32.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdebc0b6680 msgr2=0x7fdebc0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:32.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdebc0b6680 0x7fdebc0b8b90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009bd0 tx=0x7fdec8005f90 comp rx=0 tx=0).stop 2024-09-21T01:14:32.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 msgr2=0x7fded81a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:32.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 0x7fded81a0e00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdec400e9c0 tx=0x7fdec400ee80 comp rx=0 tx=0).stop 2024-09-21T01:14:32.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 shutdown_connections 2024-09-21T01:14:32.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdebc0b6680 0x7fdebc0b8b90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fded8114260 0x7fded81a60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fded8077380 0x7fded81a0e00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 --2- 172.21.15.39:0/2948068941 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fded8075dd0 0x7fded81a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:32.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 >> 172.21.15.39:0/2948068941 conn(0x7fded80ff440 msgr2=0x7fded8110160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:32.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 shutdown_connections 2024-09-21T01:14:32.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:32.488+0000 7fded7e00640 1 -- 172.21.15.39:0/2948068941 wait complete. 2024-09-21T01:14:32.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:32 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2948068941' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:33.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:32 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2948068941' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:33.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:32 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2948068941' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:14:33.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.76 MB/sec execute 420 sec latency 22877.082 ms 2024-09-21T01:14:33.453 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 97fbf2fb --release --await' 2024-09-21T01:14:33.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 97fbf2fb --release --await 2024-09-21T01:14:33.839 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:14:33.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:33 smithi039 ceph-mon[30415]: pgmap v730: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 329 KiB/s wr, 16 op/s 2024-09-21T01:14:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:33 smithi150 ceph-mon[34945]: pgmap v730: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 329 KiB/s wr, 16 op/s 2024-09-21T01:14:34.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:33 smithi160 ceph-mon[34696]: pgmap v730: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 329 KiB/s wr, 16 op/s 2024-09-21T01:14:34.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 432837 3.75 MB/sec execute 421 sec latency 23877.188 ms 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 -- 172.21.15.39:0/688697573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381092f0 msgr2=0x7f5a38109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 --2- 172.21.15.39:0/688697573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381092f0 0x7f5a38109700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009900 tx=0x7f5a2402a410 comp rx=0 tx=0).stop 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 -- 172.21.15.39:0/688697573 shutdown_connections 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 --2- 172.21.15.39:0/688697573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a38109c40 0x7f5a38110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 --2- 172.21.15.39:0/688697573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381092f0 0x7f5a38109700 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 --2- 172.21.15.39:0/688697573 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5a381039b0 0x7f5a38103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 -- 172.21.15.39:0/688697573 >> 172.21.15.39:0/688697573 conn(0x7f5a380ff400 msgr2=0x7f5a38101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 -- 172.21.15.39:0/688697573 shutdown_connections 2024-09-21T01:14:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.572+0000 7f5a3f000640 1 -- 172.21.15.39:0/688697573 wait complete. 2024-09-21T01:14:34.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.573+0000 7f5a3f000640 1 Processor -- start 2024-09-21T01:14:34.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.573+0000 7f5a3f000640 1 -- start start 2024-09-21T01:14:34.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.573+0000 7f5a3f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 0x7f5a3819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3f000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a381092f0 0x7f5a3819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3f000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5a38109c40 0x7f5a381a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5a38113900 con 0x7f5a381039b0 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3f000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5a38113780 con 0x7f5a381092f0 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3f000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5a38113a80 con 0x7f5a38109c40 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 0x7f5a3819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 0x7f5a3819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50216/0 (socket says 172.21.15.39:50216) 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 -- 172.21.15.39:0/3690951512 learned_addr learned my addr 172.21.15.39:0/3690951512 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3e600640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5a38109c40 0x7f5a381a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3d200640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a381092f0 0x7f5a3819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 -- 172.21.15.39:0/3690951512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5a38109c40 msgr2=0x7f5a381a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5a38109c40 0x7f5a381a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 -- 172.21.15.39:0/3690951512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a381092f0 msgr2=0x7f5a3819cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a381092f0 0x7f5a3819cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.574+0000 7f5a3dc00640 1 -- 172.21.15.39:0/3690951512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a381a24a0 con 0x7f5a381039b0 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.575+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 0x7f5a3819c530 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00e9f0 tx=0x7f5a2c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.575+0000 7f5a36000640 1 -- 172.21.15.39:0/3690951512 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00cd80 con 0x7f5a381039b0 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.575+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a381a2790 con 0x7f5a381039b0 2024-09-21T01:14:34.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.575+0000 7f5a36000640 1 -- 172.21.15.39:0/3690951512 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a2c0054d0 con 0x7f5a381039b0 2024-09-21T01:14:34.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.575+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a381a8ef0 con 0x7f5a381039b0 2024-09-21T01:14:34.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.575+0000 7f5a36000640 1 -- 172.21.15.39:0/3690951512 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c018630 con 0x7f5a381039b0 2024-09-21T01:14:34.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.579+0000 7f5a47269740 1 Processor -- start 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 -- start start 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ea6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ea6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bf23e1d290 con 0x55bf23aff520 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55bf23e0c6c0 con 0x55bf23ad6660 2024-09-21T01:14:34.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a47269740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55bf23e104d0 con 0x55bf23ade840 2024-09-21T01:14:34.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a3e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ea6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a3d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ea6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a3d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ea6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42838/0 (socket says 172.21.15.39:42838) 2024-09-21T01:14:34.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a3d200640 1 -- 172.21.15.39:0/3894508871 learned_addr learned my addr 172.21.15.39:0/3894508871 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:34.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.580+0000 7f5a3e600640 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ea6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50226/0 (socket says 172.21.15.39:50226) 2024-09-21T01:14:34.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a36000640 1 -- 172.21.15.39:0/3690951512 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5a2c004e70 con 0x7f5a381039b0 2024-09-21T01:14:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a3d200640 1 -- 172.21.15.39:0/3894508871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 msgr2=0x55bf23ea6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a3d200640 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a3d200640 1 -- 172.21.15.39:0/3894508871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 msgr2=0x55bf23ea6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a3d200640 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ea6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a3d200640 1 -- 172.21.15.39:0/3894508871 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf23d99460 con 0x55bf23ade840 2024-09-21T01:14:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a3d200640 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ea6dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009900 tx=0x7f5a24000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9a033a482c9183 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:34.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.581+0000 7f5a2aa00640 1 -- 172.21.15.39:0/3894508871 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a24040070 con 0x55bf23ade840 2024-09-21T01:14:34.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.582+0000 7f5a2aa00640 1 -- 172.21.15.39:0/3894508871 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a24004030 con 0x55bf23ade840 2024-09-21T01:14:34.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.582+0000 7f5a2aa00640 1 -- 172.21.15.39:0/3894508871 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a24035040 con 0x55bf23ade840 2024-09-21T01:14:34.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 -- 172.21.15.39:0/3894508871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 msgr2=0x55bf23ea6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ea6dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009900 tx=0x7f5a24000bc0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a36000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5a200b6680 0x7f5a200b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 -- 172.21.15.39:0/3894508871 shutdown_connections 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ea6dc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 --2- 172.21.15.39:0/3894508871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ea6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 -- 172.21.15.39:0/3894508871 >> 172.21.15.39:0/3894508871 conn(0x55bf23af3910 msgr2=0x55bf23ea4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a36000640 1 -- 172.21.15.39:0/3690951512 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9511+0+0 (secure 0 0 0) 0x7f5a2c0e80b0 con 0x7f5a381039b0 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a3e600640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5a200b6680 0x7f5a200b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 -- 172.21.15.39:0/3894508871 shutdown_connections 2024-09-21T01:14:34.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.583+0000 7f5a47269740 1 -- 172.21.15.39:0/3894508871 wait complete. 2024-09-21T01:14:34.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.584+0000 7f5a3e600640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5a200b6680 0x7f5a200b8b90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a30005fd0 tx=0x7f5a30009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:34.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.584+0000 7f5a47269740 1 -- start start 2024-09-21T01:14:34.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.584+0000 7f5a47269740 1 Processor -- start 2024-09-21T01:14:34.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ebadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ade840 0x55bf23ebb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 0x55bf23ec03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bf23c6dbb0 con 0x55bf23ad6660 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55bf23da76c0 con 0x55bf23ade840 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55bf23eb4ac0 con 0x55bf23aff520 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a47269740 10 client.0 _ll_register_callbacks cb 0x55bf23dc2490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a3dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ebadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a3e600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 0x55bf23ec03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a3d200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ade840 0x55bf23ebb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a3e600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 0x55bf23ec03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42850/0 (socket says 172.21.15.39:42850) 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a3dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ebadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50232/0 (socket says 172.21.15.39:50232) 2024-09-21T01:14:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.585+0000 7f5a3e600640 1 -- 172.21.15.39:0/2520469222 learned_addr learned my addr 172.21.15.39:0/2520469222 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:34.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3dc00640 1 -- 172.21.15.39:0/2520469222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 msgr2=0x55bf23ec03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 0x55bf23ec03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3dc00640 1 -- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ade840 msgr2=0x55bf23ebb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ade840 0x55bf23ebb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3dc00640 1 -- 172.21.15.39:0/2520469222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf23b6ab00 con 0x55bf23ad6660 2024-09-21T01:14:34.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3e600640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 0x55bf23ec03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3d200640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ade840 0x55bf23ebb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ebadb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0e53c0 tx=0x7f5a2c0a9a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb7dd7fa38155c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a1f400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a1f400640 1 -- 172.21.15.39:0/2520469222 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0ad230 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a1f400640 1 -- 172.21.15.39:0/2520469222 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9511+0+0 (secure 0 0 0) 0x7f5a2c0e5640 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.586+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a24004030 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a1f400640 1 -- 172.21.15.39:0/2520469222 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5a2c0d5080 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 10 client.15984 fetch_fsmap learned FSMap version 35 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bf23be1960 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a1f400640 1 -- 172.21.15.39:0/2520469222 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a24004030 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a1f400640 1 -- 172.21.15.39:0/2520469222 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f5a2c098050 con 0x55bf23ad6660 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 10 client.15984 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 10 client.15984 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 20 client.15984 populate_metadata read hostname 'smithi039' 2024-09-21T01:14:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23abdea0 0x55bf23ecc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 4 client.15984 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:14:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.587+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf23e1f1e0 con 0x55bf23abdea0 2024-09-21T01:14:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.588+0000 7f5a3d200640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23abdea0 0x55bf23ecc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.588+0000 7f5a3d200640 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23abdea0 0x55bf23ecc5a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28cccc5697faa0d server_cookie=97f2f4f23f954d01 in_seq=0 out_seq=0 2024-09-21T01:14:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.588+0000 7f5a1f400640 10 client.15984 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.589+0000 7f5a1f400640 1 -- 172.21.15.39:0/2520469222 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bf23e1f1e0 con 0x55bf23abdea0 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.589+0000 7f5a1f400640 10 client.15984 handle_command_reply: tid=0 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.589+0000 7f5a47269740 1 client.15984 shutdown 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 msgr2=0x55bf23ebadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ebadb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0e53c0 tx=0x7f5a2c0a9a20 comp rx=0 tx=0).stop 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 shutdown_connections 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23abdea0 0x55bf23ecc5a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23aff520 0x55bf23ec03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ade840 0x55bf23ebb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 --2- 172.21.15.39:0/2520469222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ebadb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 >> 172.21.15.39:0/2520469222 conn(0x55bf23af3910 msgr2=0x55bf23ea9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:34.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a1f400640 0 client.15984 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:34.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 shutdown_connections 2024-09-21T01:14:34.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 1 -- 172.21.15.39:0/2520469222 wait complete. 2024-09-21T01:14:34.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.590+0000 7f5a47269740 20 client.15984 trim_cache size 0 max 16384 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 Processor -- start 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 -- start start 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ebca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ec0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381b1460 con 0x55bf23aff520 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381aecd0 con 0x55bf23ad6660 2024-09-21T01:14:34.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.608+0000 7f5a47269740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381acbe0 con 0x55bf23ade840 2024-09-21T01:14:34.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ec0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ec0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50236/0 (socket says 172.21.15.39:50236) 2024-09-21T01:14:34.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 -- 172.21.15.39:0/82340544 learned_addr learned my addr 172.21.15.39:0/82340544 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:34.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3d200640 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ebca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 -- 172.21.15.39:0/82340544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 msgr2=0x55bf23ebca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ebca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 -- 172.21.15.39:0/82340544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 msgr2=0x55bf23ea3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.609+0000 7f5a3e600640 1 -- 172.21.15.39:0/82340544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf23eb4650 con 0x55bf23aff520 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.610+0000 7f5a3e600640 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ec0190 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5a30004360 tx=0x7f5a3001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5fb84da0fb360ff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.610+0000 7f5a2a000640 1 -- 172.21.15.39:0/82340544 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a30004b20 con 0x55bf23aff520 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.610+0000 7f5a2a000640 1 -- 172.21.15.39:0/82340544 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a30004cc0 con 0x55bf23aff520 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.610+0000 7f5a2a000640 1 -- 172.21.15.39:0/82340544 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a300057c0 con 0x55bf23aff520 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 -- 172.21.15.39:0/82340544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 msgr2=0x55bf23ec0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ec0190 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5a30004360 tx=0x7f5a3001a000 comp rx=0 tx=0).stop 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 -- 172.21.15.39:0/82340544 shutdown_connections 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23aff520 0x55bf23ec0190 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ebca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 --2- 172.21.15.39:0/82340544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23ad6660 0x55bf23ea3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 -- 172.21.15.39:0/82340544 >> 172.21.15.39:0/82340544 conn(0x55bf23af3910 msgr2=0x55bf23ea41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 -- 172.21.15.39:0/82340544 shutdown_connections 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 -- 172.21.15.39:0/82340544 wait complete. 2024-09-21T01:14:34.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.611+0000 7f5a47269740 1 -- start start 2024-09-21T01:14:34.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 Processor -- start 2024-09-21T01:14:34.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ec15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ec1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23aff520 0x55bf23ec6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381ad230 con 0x55bf23ad6660 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381ae600 con 0x55bf23aff520 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381ad880 con 0x55bf23ade840 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a47269740 10 client.0 _ll_register_callbacks cb 0x7f5a2c004990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a3dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ec15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a3dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ec15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50242/0 (socket says 172.21.15.39:50242) 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a3d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ec1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a3dc00640 1 -- 172.21.15.39:0/1504011223 learned_addr learned my addr 172.21.15.39:0/1504011223 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.612+0000 7f5a3e600640 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23aff520 0x55bf23ec6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a3dc00640 1 -- 172.21.15.39:0/1504011223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 msgr2=0x55bf23ec1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ec1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a3dc00640 1 -- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23aff520 msgr2=0x55bf23ec6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:14:34.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23aff520 0x55bf23ec6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a3dc00640 1 -- 172.21.15.39:0/1504011223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf23eaa450 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a3dc00640 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ec15e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0b2690 tx=0x7f5a2c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6999bf3434853d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a1f400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a1f400640 1 -- 172.21.15.39:0/1504011223 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0c7070 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a1f400640 1 -- 172.21.15.39:0/1504011223 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9511+0+0 (secure 0 0 0) 0x7f5a2c0e5d10 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.613+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a30004cc0 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a1f400640 1 -- 172.21.15.39:0/1504011223 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5a2c0e5850 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 10 client.15990 fetch_fsmap learned FSMap version 35 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bf23f37520 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a1f400640 1 -- 172.21.15.39:0/1504011223 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a30004cc0 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a1f400640 1 -- 172.21.15.39:0/1504011223 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f5a2c098070 con 0x55bf23ad6660 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 10 client.15990 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 10 client.15990 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 20 client.15990 populate_metadata read hostname 'smithi039' 2024-09-21T01:14:34.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23a93250 0x55bf23f39b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:14:34.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 4 client.15990 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "97fbf2fb", "release": true, "await": true}] 2024-09-21T01:14:34.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.614+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "97fbf2fb", "release": true, "await": true}) -- 0x55bf23e1ac90 con 0x55bf23a93250 2024-09-21T01:14:34.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.615+0000 7f5a3d200640 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23a93250 0x55bf23f39b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:14:34.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.615+0000 7f5a3d200640 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23a93250 0x55bf23f39b80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89d7e5d253053da3 server_cookie=58312ec6a928bb96 in_seq=0 out_seq=0 2024-09-21T01:14:34.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.615+0000 7f5a1f400640 10 client.15990 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:34.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a1f400640 1 -- 172.21.15.39:0/1504011223 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55bf23e1ac90 con 0x55bf23a93250 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a1f400640 10 client.15990 handle_command_reply: tid=0 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 client.15990 shutdown 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 msgr2=0x55bf23ec15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ec15e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0b2690 tx=0x7f5a2c00cf20 comp rx=0 tx=0).stop 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 shutdown_connections 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55bf23a93250 0x55bf23f39b80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bf23aff520 0x55bf23ec6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55bf23ade840 0x55bf23ec1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 --2- 172.21.15.39:0/1504011223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf23ad6660 0x55bf23ec15e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 >> 172.21.15.39:0/1504011223 conn(0x55bf23af3910 msgr2=0x55bf23eb9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.621+0000 7f5a1f400640 0 client.15990 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:14:34.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 shutdown_connections 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a47269740 1 -- 172.21.15.39:0/1504011223 wait complete. 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a47269740 20 client.15990 trim_cache size 0 max 16384 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 35, 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 44, 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: "97fbf2fb": { 2024-09-21T01:14:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 44, 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 144.5, 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:14:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:14:34.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5a200b6680 msgr2=0x7f5a200b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a3f000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5a200b6680 0x7f5a200b8b90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a30005fd0 tx=0x7f5a30009500 comp rx=0 tx=0).stop 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 msgr2=0x7f5a3819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.622+0000 7f5a3f000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 0x7f5a3819c530 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00e9f0 tx=0x7f5a2c00edf0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 shutdown_connections 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5a38109c40 0x7f5a381a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5a200b6680 0x7f5a200b8b90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a381092f0 0x7f5a3819cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 --2- 172.21.15.39:0/3690951512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a381039b0 0x7f5a3819c530 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 >> 172.21.15.39:0/3690951512 conn(0x7f5a380ff400 msgr2=0x7f5a38106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 shutdown_connections 2024-09-21T01:14:34.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:14:34.623+0000 7f5a3f000640 1 -- 172.21.15.39:0/3690951512 wait complete. 2024-09-21T01:14:35.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 433386 3.75 MB/sec execute 422 sec latency 24310.725 ms 2024-09-21T01:14:35.555 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '97fbf2fb', total seconds elapsed: 28.9 2024-09-21T01:14:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:35 smithi039 ceph-mon[30415]: pgmap v731: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 329 KiB/s wr, 17 op/s 2024-09-21T01:14:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:35 smithi150 ceph-mon[34945]: pgmap v731: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 329 KiB/s wr, 17 op/s 2024-09-21T01:14:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:35 smithi160 ceph-mon[34696]: pgmap v731: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 329 KiB/s wr, 17 op/s 2024-09-21T01:14:36.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 434458 3.75 MB/sec execute 423 sec latency 240.990 ms 2024-09-21T01:14:37.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 434458 3.74 MB/sec execute 424 sec latency 1241.118 ms 2024-09-21T01:14:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:37 smithi150 ceph-mon[34945]: pgmap v732: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 342 KiB/s wr, 17 op/s 2024-09-21T01:14:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:37 smithi160 ceph-mon[34696]: pgmap v732: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 342 KiB/s wr, 17 op/s 2024-09-21T01:14:38.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:37 smithi039 ceph-mon[30415]: pgmap v732: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 342 KiB/s wr, 17 op/s 2024-09-21T01:14:38.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 434458 3.73 MB/sec execute 425 sec latency 2241.189 ms 2024-09-21T01:14:39.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:38 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:38 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:39.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:38 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:39.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 434458 3.72 MB/sec execute 426 sec latency 3241.241 ms 2024-09-21T01:14:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:39 smithi150 ceph-mon[34945]: pgmap v733: 129 pgs: 129 active+clean; 306 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 412 KiB/s wr, 30 op/s 2024-09-21T01:14:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:39 smithi160 ceph-mon[34696]: pgmap v733: 129 pgs: 129 active+clean; 306 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 412 KiB/s wr, 30 op/s 2024-09-21T01:14:40.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:39 smithi039 ceph-mon[30415]: pgmap v733: 129 pgs: 129 active+clean; 306 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 412 KiB/s wr, 30 op/s 2024-09-21T01:14:40.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 436036 3.74 MB/sec execute 427 sec latency 3588.342 ms 2024-09-21T01:14:41.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 437306 3.74 MB/sec execute 428 sec latency 20.045 ms 2024-09-21T01:14:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:41 smithi150 ceph-mon[34945]: pgmap v734: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 388 KiB/s wr, 37 op/s 2024-09-21T01:14:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:41 smithi160 ceph-mon[34696]: pgmap v734: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 388 KiB/s wr, 37 op/s 2024-09-21T01:14:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:41 smithi039 ceph-mon[30415]: pgmap v734: 129 pgs: 129 active+clean; 307 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 388 KiB/s wr, 37 op/s 2024-09-21T01:14:42.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 438514 3.74 MB/sec execute 429 sec latency 366.694 ms 2024-09-21T01:14:43.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 440382 3.77 MB/sec execute 430 sec latency 11.948 ms 2024-09-21T01:14:44.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:43 smithi150 ceph-mon[34945]: pgmap v735: 129 pgs: 129 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 311 KiB/s wr, 39 op/s 2024-09-21T01:14:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:43 smithi160 ceph-mon[34696]: pgmap v735: 129 pgs: 129 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 311 KiB/s wr, 39 op/s 2024-09-21T01:14:44.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:43 smithi039 ceph-mon[30415]: pgmap v735: 129 pgs: 129 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 311 KiB/s wr, 39 op/s 2024-09-21T01:14:44.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 441504 3.77 MB/sec execute 431 sec latency 16.469 ms 2024-09-21T01:14:45.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 443413 3.79 MB/sec execute 432 sec latency 158.840 ms 2024-09-21T01:14:46.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:45 smithi150 ceph-mon[34945]: pgmap v736: 129 pgs: 129 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 382 KiB/s wr, 52 op/s 2024-09-21T01:14:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:45 smithi160 ceph-mon[34696]: pgmap v736: 129 pgs: 129 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 382 KiB/s wr, 52 op/s 2024-09-21T01:14:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:45 smithi039 ceph-mon[30415]: pgmap v736: 129 pgs: 129 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 382 KiB/s wr, 52 op/s 2024-09-21T01:14:46.189 INFO:tasks.workunit.client.0.smithi039.stdout: 1 444636 3.79 MB/sec execute 433 sec latency 17.417 ms 2024-09-21T01:14:47.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 445096 3.79 MB/sec execute 434 sec latency 556.032 ms 2024-09-21T01:14:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:47 smithi150 ceph-mon[34945]: pgmap v737: 129 pgs: 129 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 701 KiB/s wr, 75 op/s 2024-09-21T01:14:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:47 smithi160 ceph-mon[34696]: pgmap v737: 129 pgs: 129 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 701 KiB/s wr, 75 op/s 2024-09-21T01:14:48.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:47 smithi039 ceph-mon[30415]: pgmap v737: 129 pgs: 129 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 701 KiB/s wr, 75 op/s 2024-09-21T01:14:48.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 445096 3.78 MB/sec execute 435 sec latency 1556.151 ms 2024-09-21T01:14:49.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 445096 3.77 MB/sec execute 436 sec latency 2556.269 ms 2024-09-21T01:14:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:49 smithi150 ceph-mon[34945]: pgmap v738: 129 pgs: 129 active+clean; 319 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-09-21T01:14:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:49 smithi160 ceph-mon[34696]: pgmap v738: 129 pgs: 129 active+clean; 319 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-09-21T01:14:50.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:49 smithi039 ceph-mon[30415]: pgmap v738: 129 pgs: 129 active+clean; 319 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-09-21T01:14:50.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 447020 3.79 MB/sec execute 437 sec latency 2745.653 ms 2024-09-21T01:14:51.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 448184 3.79 MB/sec execute 438 sec latency 18.424 ms 2024-09-21T01:14:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:51 smithi150 ceph-mon[34945]: pgmap v739: 129 pgs: 129 active+clean; 319 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-09-21T01:14:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:51 smithi160 ceph-mon[34696]: pgmap v739: 129 pgs: 129 active+clean; 319 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-09-21T01:14:52.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:51 smithi039 ceph-mon[30415]: pgmap v739: 129 pgs: 129 active+clean; 319 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-09-21T01:14:52.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 449146 3.79 MB/sec execute 439 sec latency 381.269 ms 2024-09-21T01:14:53.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 451034 3.81 MB/sec execute 440 sec latency 11.986 ms 2024-09-21T01:14:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:53 smithi150 ceph-mon[34945]: pgmap v740: 129 pgs: 129 active+clean; 322 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T01:14:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:53 smithi160 ceph-mon[34696]: pgmap v740: 129 pgs: 129 active+clean; 322 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T01:14:54.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:53 smithi039 ceph-mon[30415]: pgmap v740: 129 pgs: 129 active+clean; 322 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-09-21T01:14:54.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 452163 3.81 MB/sec execute 441 sec latency 16.304 ms 2024-09-21T01:14:55.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 454038 3.83 MB/sec execute 442 sec latency 183.651 ms 2024-09-21T01:14:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:55 smithi150 ceph-mon[34945]: pgmap v741: 129 pgs: 129 active+clean; 324 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T01:14:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:55 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:55 smithi160 ceph-mon[34696]: pgmap v741: 129 pgs: 129 active+clean; 324 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T01:14:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:55 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:55 smithi039 ceph-mon[30415]: pgmap v741: 129 pgs: 129 active+clean; 324 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 144 op/s 2024-09-21T01:14:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:55 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:14:56.190 INFO:tasks.workunit.client.0.smithi039.stdout: 1 455207 3.83 MB/sec execute 443 sec latency 18.612 ms 2024-09-21T01:14:57.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 456239 3.83 MB/sec execute 444 sec latency 304.188 ms 2024-09-21T01:14:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:57 smithi150 ceph-mon[34945]: pgmap v742: 129 pgs: 129 active+clean; 324 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-09-21T01:14:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:57 smithi160 ceph-mon[34696]: pgmap v742: 129 pgs: 129 active+clean; 324 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-09-21T01:14:58.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:57 smithi039 ceph-mon[30415]: pgmap v742: 129 pgs: 129 active+clean; 324 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-09-21T01:14:58.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 458054 3.85 MB/sec execute 445 sec latency 19.412 ms 2024-09-21T01:14:59.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 458837 3.88 MB/sec execute 446 sec latency 44.535 ms 2024-09-21T01:15:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:14:59 smithi150 ceph-mon[34945]: pgmap v743: 129 pgs: 129 active+clean; 331 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-09-21T01:15:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:14:59 smithi160 ceph-mon[34696]: pgmap v743: 129 pgs: 129 active+clean; 331 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-09-21T01:15:00.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:14:59 smithi039 ceph-mon[30415]: pgmap v743: 129 pgs: 129 active+clean; 331 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-09-21T01:15:00.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 460525 3.90 MB/sec execute 447 sec latency 24.555 ms 2024-09-21T01:15:01.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:00 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:01.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:00 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:01.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:00 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:01.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461246 3.90 MB/sec execute 448 sec latency 707.196 ms 2024-09-21T01:15:02.027 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:01 smithi039 ceph-mon[30415]: pgmap v744: 129 pgs: 129 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-21T01:15:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:01 smithi150 ceph-mon[34945]: pgmap v744: 129 pgs: 129 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-21T01:15:02.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:01 smithi160 ceph-mon[34696]: pgmap v744: 129 pgs: 129 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-09-21T01:15:02.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461319 3.90 MB/sec execute 449 sec latency 1510.051 ms 2024-09-21T01:15:03.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461319 3.89 MB/sec execute 450 sec latency 1110.457 ms 2024-09-21T01:15:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:03 smithi150 ceph-mon[34945]: pgmap v745: 129 pgs: 129 active+clean; 336 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T01:15:04.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:03 smithi160 ceph-mon[34696]: pgmap v745: 129 pgs: 129 active+clean; 336 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T01:15:04.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:03 smithi039 ceph-mon[30415]: pgmap v745: 129 pgs: 129 active+clean; 336 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 153 op/s 2024-09-21T01:15:04.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461319 3.88 MB/sec execute 451 sec latency 2110.605 ms 2024-09-21T01:15:05.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461520 3.87 MB/sec execute 452 sec latency 2298.649 ms 2024-09-21T01:15:05.977 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:05 smithi039 ceph-mon[30415]: pgmap v746: 129 pgs: 129 active+clean; 336 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-09-21T01:15:05.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:05 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:05.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:05 smithi039 ceph-mon[30415]: osdmap e101: 12 total, 12 up, 12 in 2024-09-21T01:15:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:05 smithi150 ceph-mon[34945]: pgmap v746: 129 pgs: 129 active+clean; 336 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-09-21T01:15:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:05 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:05 smithi150 ceph-mon[34945]: osdmap e101: 12 total, 12 up, 12 in 2024-09-21T01:15:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:05 smithi160 ceph-mon[34696]: pgmap v746: 129 pgs: 129 active+clean; 336 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-09-21T01:15:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:05 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:05 smithi160 ceph-mon[34696]: osdmap e101: 12 total, 12 up, 12 in 2024-09-21T01:15:06.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461520 3.86 MB/sec execute 453 sec latency 1619.950 ms 2024-09-21T01:15:07.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461520 3.85 MB/sec execute 454 sec latency 2620.000 ms 2024-09-21T01:15:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:07 smithi150 ceph-mon[34945]: pgmap v748: 129 pgs: 129 active+clean; 338 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 172 op/s 2024-09-21T01:15:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:07 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:15:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:07 smithi160 ceph-mon[34696]: pgmap v748: 129 pgs: 129 active+clean; 338 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 172 op/s 2024-09-21T01:15:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:07 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:15:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:07 smithi039 ceph-mon[30415]: pgmap v748: 129 pgs: 129 active+clean; 338 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 172 op/s 2024-09-21T01:15:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:07 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:15:08.191 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461520 3.85 MB/sec execute 455 sec latency 3620.079 ms 2024-09-21T01:15:09.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461520 3.84 MB/sec execute 456 sec latency 4620.122 ms 2024-09-21T01:15:09.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:09 smithi039 ceph-mon[30415]: pgmap v749: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 4.5 MiB/s wr, 157 op/s 2024-09-21T01:15:09.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:15:09.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:15:09.898 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:09 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:15:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:09 smithi150 ceph-mon[34945]: pgmap v749: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 4.5 MiB/s wr, 157 op/s 2024-09-21T01:15:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:15:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:15:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:09 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:15:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:09 smithi160 ceph-mon[34696]: pgmap v749: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 4.5 MiB/s wr, 157 op/s 2024-09-21T01:15:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:15:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:15:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:09 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:15:10.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461553 3.83 MB/sec execute 457 sec latency 5531.619 ms 2024-09-21T01:15:11.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461553 3.82 MB/sec execute 458 sec latency 1066.805 ms 2024-09-21T01:15:12.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:11 smithi150 ceph-mon[34945]: pgmap v750: 129 pgs: 129 active+clean; 373 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 5.3 MiB/s wr, 148 op/s 2024-09-21T01:15:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:11 smithi160 ceph-mon[34696]: pgmap v750: 129 pgs: 129 active+clean; 373 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 5.3 MiB/s wr, 148 op/s 2024-09-21T01:15:12.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:11 smithi039 ceph-mon[30415]: pgmap v750: 129 pgs: 129 active+clean; 373 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 5.3 MiB/s wr, 148 op/s 2024-09-21T01:15:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461696 3.81 MB/sec execute 459 sec latency 1889.686 ms 2024-09-21T01:15:13.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461696 3.81 MB/sec execute 460 sec latency 1130.529 ms 2024-09-21T01:15:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:13 smithi160 ceph-mon[34696]: pgmap v751: 129 pgs: 129 active+clean; 364 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.8 MiB/s wr, 132 op/s 2024-09-21T01:15:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:13 smithi039 ceph-mon[30415]: pgmap v751: 129 pgs: 129 active+clean; 364 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.8 MiB/s wr, 132 op/s 2024-09-21T01:15:14.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 461696 3.80 MB/sec execute 461 sec latency 2130.674 ms 2024-09-21T01:15:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:13 smithi150 ceph-mon[34945]: pgmap v751: 129 pgs: 129 active+clean; 364 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.8 MiB/s wr, 132 op/s 2024-09-21T01:15:15.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 462479 3.80 MB/sec execute 462 sec latency 2316.321 ms 2024-09-21T01:15:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:15 smithi150 ceph-mon[34945]: pgmap v752: 129 pgs: 129 active+clean; 357 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 4.8 MiB/s wr, 116 op/s 2024-09-21T01:15:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:15 smithi150 ceph-mon[34945]: osdmap e102: 12 total, 12 up, 12 in 2024-09-21T01:15:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:15 smithi160 ceph-mon[34696]: pgmap v752: 129 pgs: 129 active+clean; 357 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 4.8 MiB/s wr, 116 op/s 2024-09-21T01:15:15.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:15 smithi160 ceph-mon[34696]: osdmap e102: 12 total, 12 up, 12 in 2024-09-21T01:15:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:15 smithi039 ceph-mon[30415]: pgmap v752: 129 pgs: 129 active+clean; 357 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 4.8 MiB/s wr, 116 op/s 2024-09-21T01:15:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:15 smithi039 ceph-mon[30415]: osdmap e102: 12 total, 12 up, 12 in 2024-09-21T01:15:16.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 464203 3.81 MB/sec execute 463 sec latency 19.066 ms 2024-09-21T01:15:17.192 INFO:tasks.workunit.client.0.smithi039.stdout: 1 466039 3.82 MB/sec execute 464 sec latency 13.542 ms 2024-09-21T01:15:17.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:17 smithi150 ceph-mon[34945]: pgmap v754: 129 pgs: 129 active+clean; 357 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 4.7 MiB/s wr, 111 op/s 2024-09-21T01:15:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:17 smithi160 ceph-mon[34696]: pgmap v754: 129 pgs: 129 active+clean; 357 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 4.7 MiB/s wr, 111 op/s 2024-09-21T01:15:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:17 smithi039 ceph-mon[30415]: pgmap v754: 129 pgs: 129 active+clean; 357 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 4.7 MiB/s wr, 111 op/s 2024-09-21T01:15:18.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 467888 3.83 MB/sec execute 465 sec latency 16.130 ms 2024-09-21T01:15:19.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 469714 3.85 MB/sec execute 466 sec latency 20.055 ms 2024-09-21T01:15:19.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:19 smithi150 ceph-mon[34945]: pgmap v755: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.3 MiB/s wr, 64 op/s 2024-09-21T01:15:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:19 smithi160 ceph-mon[34696]: pgmap v755: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.3 MiB/s wr, 64 op/s 2024-09-21T01:15:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:19 smithi039 ceph-mon[30415]: pgmap v755: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.3 MiB/s wr, 64 op/s 2024-09-21T01:15:20.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 471592 3.86 MB/sec execute 467 sec latency 14.507 ms 2024-09-21T01:15:21.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 473272 3.87 MB/sec execute 468 sec latency 13.477 ms 2024-09-21T01:15:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:21 smithi150 ceph-mon[34945]: pgmap v756: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 842 KiB/s wr, 98 op/s 2024-09-21T01:15:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:21 smithi160 ceph-mon[34696]: pgmap v756: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 842 KiB/s wr, 98 op/s 2024-09-21T01:15:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:21 smithi039 ceph-mon[30415]: pgmap v756: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 842 KiB/s wr, 98 op/s 2024-09-21T01:15:22.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 474987 3.88 MB/sec execute 469 sec latency 39.826 ms 2024-09-21T01:15:23.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 476776 3.90 MB/sec execute 470 sec latency 19.076 ms 2024-09-21T01:15:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:23 smithi150 ceph-mon[34945]: pgmap v757: 129 pgs: 129 active+clean; 367 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 1.2 MiB/s wr, 106 op/s 2024-09-21T01:15:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:23 smithi160 ceph-mon[34696]: pgmap v757: 129 pgs: 129 active+clean; 367 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 1.2 MiB/s wr, 106 op/s 2024-09-21T01:15:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:23 smithi039 ceph-mon[30415]: pgmap v757: 129 pgs: 129 active+clean; 367 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 1.2 MiB/s wr, 106 op/s 2024-09-21T01:15:24.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 478630 3.91 MB/sec execute 471 sec latency 13.414 ms 2024-09-21T01:15:25.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 480288 3.92 MB/sec execute 472 sec latency 22.969 ms 2024-09-21T01:15:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:25 smithi150 ceph-mon[34945]: pgmap v758: 129 pgs: 129 active+clean; 371 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-09-21T01:15:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:25 smithi160 ceph-mon[34696]: pgmap v758: 129 pgs: 129 active+clean; 371 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-09-21T01:15:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:25 smithi039 ceph-mon[30415]: pgmap v758: 129 pgs: 129 active+clean; 371 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-09-21T01:15:26.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 482086 3.93 MB/sec execute 473 sec latency 34.498 ms 2024-09-21T01:15:27.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 483765 3.95 MB/sec execute 474 sec latency 13.006 ms 2024-09-21T01:15:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:27 smithi150 ceph-mon[34945]: pgmap v759: 129 pgs: 129 active+clean; 376 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 149 op/s 2024-09-21T01:15:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:27 smithi160 ceph-mon[34696]: pgmap v759: 129 pgs: 129 active+clean; 376 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 149 op/s 2024-09-21T01:15:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:27 smithi039 ceph-mon[30415]: pgmap v759: 129 pgs: 129 active+clean; 376 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 149 op/s 2024-09-21T01:15:28.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 485198 3.95 MB/sec execute 475 sec latency 19.868 ms 2024-09-21T01:15:29.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 487138 3.97 MB/sec execute 476 sec latency 16.040 ms 2024-09-21T01:15:29.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:29 smithi150 ceph-mon[34945]: pgmap v760: 129 pgs: 129 active+clean; 381 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-09-21T01:15:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:29 smithi160 ceph-mon[34696]: pgmap v760: 129 pgs: 129 active+clean; 381 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-09-21T01:15:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:29 smithi039 ceph-mon[30415]: pgmap v760: 129 pgs: 129 active+clean; 381 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-09-21T01:15:30.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 488420 3.97 MB/sec execute 477 sec latency 26.926 ms 2024-09-21T01:15:31.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 489734 3.98 MB/sec execute 478 sec latency 254.034 ms 2024-09-21T01:15:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:31 smithi150 ceph-mon[34945]: pgmap v761: 129 pgs: 129 active+clean; 383 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 184 op/s 2024-09-21T01:15:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:31 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:31 smithi160 ceph-mon[34696]: pgmap v761: 129 pgs: 129 active+clean; 383 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 184 op/s 2024-09-21T01:15:31.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:31 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:31 smithi039 ceph-mon[30415]: pgmap v761: 129 pgs: 129 active+clean; 383 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 184 op/s 2024-09-21T01:15:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:31 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:15:32.193 INFO:tasks.workunit.client.0.smithi039.stdout: 1 489968 3.97 MB/sec execute 479 sec latency 1071.324 ms 2024-09-21T01:15:33.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 489968 3.96 MB/sec execute 480 sec latency 1048.329 ms 2024-09-21T01:15:33.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:33 smithi150 ceph-mon[34945]: pgmap v762: 129 pgs: 129 active+clean; 385 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 153 op/s 2024-09-21T01:15:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:33 smithi160 ceph-mon[34696]: pgmap v762: 129 pgs: 129 active+clean; 385 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 153 op/s 2024-09-21T01:15:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:33 smithi039 ceph-mon[30415]: pgmap v762: 129 pgs: 129 active+clean; 385 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 153 op/s 2024-09-21T01:15:34.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 489968 3.96 MB/sec execute 481 sec latency 2048.451 ms 2024-09-21T01:15:35.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 489968 3.95 MB/sec execute 482 sec latency 3048.592 ms 2024-09-21T01:15:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:35 smithi150 ceph-mon[34945]: pgmap v763: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 157 op/s 2024-09-21T01:15:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:35 smithi160 ceph-mon[34696]: pgmap v763: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 157 op/s 2024-09-21T01:15:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:35 smithi039 ceph-mon[30415]: pgmap v763: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 157 op/s 2024-09-21T01:15:36.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 489968 3.94 MB/sec execute 483 sec latency 4048.682 ms 2024-09-21T01:15:37.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490379 3.94 MB/sec execute 484 sec latency 4870.119 ms 2024-09-21T01:15:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:37 smithi150 ceph-mon[34945]: pgmap v764: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 140 op/s 2024-09-21T01:15:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:37 smithi160 ceph-mon[34696]: pgmap v764: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 140 op/s 2024-09-21T01:15:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:37 smithi039 ceph-mon[30415]: pgmap v764: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 140 op/s 2024-09-21T01:15:38.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490551 3.93 MB/sec execute 485 sec latency 812.195 ms 2024-09-21T01:15:39.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490551 3.93 MB/sec execute 486 sec latency 1812.334 ms 2024-09-21T01:15:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:39 smithi150 ceph-mon[34945]: pgmap v765: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-09-21T01:15:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:39 smithi160 ceph-mon[34696]: pgmap v765: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-09-21T01:15:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:39 smithi039 ceph-mon[30415]: pgmap v765: 129 pgs: 129 active+clean; 389 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-09-21T01:15:40.194 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.92 MB/sec execute 487 sec latency 2001.125 ms 2024-09-21T01:15:41.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.91 MB/sec execute 488 sec latency 1589.605 ms 2024-09-21T01:15:41.562 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:41 smithi039 ceph-mon[30415]: pgmap v766: 129 pgs: 129 active+clean; 390 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 854 KiB/s wr, 80 op/s 2024-09-21T01:15:41.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:41 smithi150 ceph-mon[34945]: pgmap v766: 129 pgs: 129 active+clean; 390 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 854 KiB/s wr, 80 op/s 2024-09-21T01:15:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:41 smithi160 ceph-mon[34696]: pgmap v766: 129 pgs: 129 active+clean; 390 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 854 KiB/s wr, 80 op/s 2024-09-21T01:15:42.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.90 MB/sec execute 489 sec latency 2589.701 ms 2024-09-21T01:15:43.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.89 MB/sec execute 490 sec latency 3589.766 ms 2024-09-21T01:15:43.533 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:43 smithi039 ceph-mon[30415]: pgmap v767: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.4 MiB/s wr, 74 op/s 2024-09-21T01:15:43.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:43 smithi150 ceph-mon[34945]: pgmap v767: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.4 MiB/s wr, 74 op/s 2024-09-21T01:15:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:43 smithi160 ceph-mon[34696]: pgmap v767: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.4 MiB/s wr, 74 op/s 2024-09-21T01:15:44.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.89 MB/sec execute 491 sec latency 4589.807 ms 2024-09-21T01:15:45.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.88 MB/sec execute 492 sec latency 5589.854 ms 2024-09-21T01:15:45.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:45 smithi039 ceph-mon[30415]: pgmap v768: 129 pgs: 129 active+clean; 405 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 2.3 MiB/s wr, 90 op/s 2024-09-21T01:15:45.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:45 smithi150 ceph-mon[34945]: pgmap v768: 129 pgs: 129 active+clean; 405 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 2.3 MiB/s wr, 90 op/s 2024-09-21T01:15:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:45 smithi160 ceph-mon[34696]: pgmap v768: 129 pgs: 129 active+clean; 405 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 2.3 MiB/s wr, 90 op/s 2024-09-21T01:15:46.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 490694 3.87 MB/sec execute 493 sec latency 6589.896 ms 2024-09-21T01:15:47.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 491111 3.86 MB/sec execute 494 sec latency 7132.079 ms 2024-09-21T01:15:47.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:47 smithi150 ceph-mon[34945]: pgmap v769: 129 pgs: 129 active+clean; 407 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 2.2 MiB/s wr, 82 op/s 2024-09-21T01:15:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:47 smithi160 ceph-mon[34696]: pgmap v769: 129 pgs: 129 active+clean; 407 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 2.2 MiB/s wr, 82 op/s 2024-09-21T01:15:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:47 smithi039 ceph-mon[30415]: pgmap v769: 129 pgs: 129 active+clean; 407 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 2.2 MiB/s wr, 82 op/s 2024-09-21T01:15:48.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 492910 3.87 MB/sec execute 495 sec latency 24.145 ms 2024-09-21T01:15:49.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 494274 3.89 MB/sec execute 496 sec latency 17.408 ms 2024-09-21T01:15:49.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:49 smithi150 ceph-mon[34945]: pgmap v770: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-09-21T01:15:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:49 smithi160 ceph-mon[34696]: pgmap v770: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-09-21T01:15:49.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:49 smithi039 ceph-mon[30415]: pgmap v770: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-09-21T01:15:50.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 495511 3.89 MB/sec execute 497 sec latency 20.206 ms 2024-09-21T01:15:51.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 497631 3.91 MB/sec execute 498 sec latency 13.533 ms 2024-09-21T01:15:51.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:51 smithi150 ceph-mon[34945]: pgmap v771: 129 pgs: 129 active+clean; 401 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 2.8 MiB/s wr, 131 op/s 2024-09-21T01:15:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:51 smithi160 ceph-mon[34696]: pgmap v771: 129 pgs: 129 active+clean; 401 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 2.8 MiB/s wr, 131 op/s 2024-09-21T01:15:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:51 smithi039 ceph-mon[30415]: pgmap v771: 129 pgs: 129 active+clean; 401 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 2.8 MiB/s wr, 131 op/s 2024-09-21T01:15:52.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 498586 3.91 MB/sec execute 499 sec latency 18.592 ms 2024-09-21T01:15:53.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 500422 3.92 MB/sec execute 500 sec latency 13.739 ms 2024-09-21T01:15:53.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:53 smithi150 ceph-mon[34945]: pgmap v772: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.8 MiB/s wr, 130 op/s 2024-09-21T01:15:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:53 smithi160 ceph-mon[34696]: pgmap v772: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.8 MiB/s wr, 130 op/s 2024-09-21T01:15:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:53 smithi039 ceph-mon[30415]: pgmap v772: 129 pgs: 129 active+clean; 398 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 2.8 MiB/s wr, 130 op/s 2024-09-21T01:15:54.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 501606 3.92 MB/sec execute 501 sec latency 22.830 ms 2024-09-21T01:15:55.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 503376 3.93 MB/sec execute 502 sec latency 20.318 ms 2024-09-21T01:15:55.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:55 smithi150 ceph-mon[34945]: pgmap v773: 129 pgs: 129 active+clean; 396 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-09-21T01:15:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:55 smithi160 ceph-mon[34696]: pgmap v773: 129 pgs: 129 active+clean; 396 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-09-21T01:15:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:55 smithi039 ceph-mon[30415]: pgmap v773: 129 pgs: 129 active+clean; 396 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-09-21T01:15:56.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 504915 3.95 MB/sec execute 503 sec latency 12.274 ms 2024-09-21T01:15:57.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 506010 3.95 MB/sec execute 504 sec latency 16.222 ms 2024-09-21T01:15:57.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:57 smithi039 ceph-mon[30415]: pgmap v774: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-09-21T01:15:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:57 smithi150 ceph-mon[34945]: pgmap v774: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-09-21T01:15:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:57 smithi160 ceph-mon[34696]: pgmap v774: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-09-21T01:15:58.195 INFO:tasks.workunit.client.0.smithi039.stdout: 1 506645 3.94 MB/sec execute 505 sec latency 602.525 ms 2024-09-21T01:15:59.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 508343 3.96 MB/sec execute 506 sec latency 89.438 ms 2024-09-21T01:15:59.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:15:59 smithi150 ceph-mon[34945]: pgmap v775: 129 pgs: 129 active+clean; 407 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 171 op/s 2024-09-21T01:15:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:15:59 smithi160 ceph-mon[34696]: pgmap v775: 129 pgs: 129 active+clean; 407 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 171 op/s 2024-09-21T01:15:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:15:59 smithi039 ceph-mon[30415]: pgmap v775: 129 pgs: 129 active+clean; 407 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 171 op/s 2024-09-21T01:16:00.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 509436 3.96 MB/sec execute 507 sec latency 19.222 ms 2024-09-21T01:16:01.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 510320 3.96 MB/sec execute 508 sec latency 84.750 ms 2024-09-21T01:16:01.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:01 smithi150 ceph-mon[34945]: pgmap v776: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-09-21T01:16:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:01 smithi160 ceph-mon[34696]: pgmap v776: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-09-21T01:16:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:01 smithi039 ceph-mon[30415]: pgmap v776: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-09-21T01:16:02.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 511568 3.97 MB/sec execute 509 sec latency 57.573 ms 2024-09-21T01:16:03.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 512381 3.97 MB/sec execute 510 sec latency 16.963 ms 2024-09-21T01:16:03.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:03 smithi150 ceph-mon[34945]: pgmap v777: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-09-21T01:16:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:03 smithi160 ceph-mon[34696]: pgmap v777: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-09-21T01:16:03.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:03 smithi039 ceph-mon[30415]: pgmap v777: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-09-21T01:16:04.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 513376 3.97 MB/sec execute 511 sec latency 117.667 ms 2024-09-21T01:16:05.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 513784 3.96 MB/sec execute 512 sec latency 538.806 ms 2024-09-21T01:16:05.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:05 smithi150 ceph-mon[34945]: pgmap v778: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 241 op/s 2024-09-21T01:16:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:05 smithi150 ceph-mon[34945]: osdmap e103: 12 total, 12 up, 12 in 2024-09-21T01:16:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:05 smithi160 ceph-mon[34696]: pgmap v778: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 241 op/s 2024-09-21T01:16:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:05 smithi160 ceph-mon[34696]: osdmap e103: 12 total, 12 up, 12 in 2024-09-21T01:16:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:05 smithi039 ceph-mon[30415]: pgmap v778: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 241 op/s 2024-09-21T01:16:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:05 smithi039 ceph-mon[30415]: osdmap e103: 12 total, 12 up, 12 in 2024-09-21T01:16:06.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 515412 3.98 MB/sec execute 513 sec latency 33.511 ms 2024-09-21T01:16:06.556 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:16:06.556 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.8 2024-09-21T01:16:06.556 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:16:06.984 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:07.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 516375 3.98 MB/sec execute 514 sec latency 15.191 ms 2024-09-21T01:16:07.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:07 smithi150 ceph-mon[34945]: pgmap v780: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 250 op/s 2024-09-21T01:16:07.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 -- 172.21.15.39:0/1296376422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e401049d0 msgr2=0x7f9e40104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:07.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 --2- 172.21.15.39:0/1296376422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e401049d0 0x7f9e40104de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009900 tx=0x7f9e3002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:16:07.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 -- 172.21.15.39:0/1296376422 shutdown_connections 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 --2- 172.21.15.39:0/1296376422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e4010de30 0x7f9e40110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 --2- 172.21.15.39:0/1296376422 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e40105320 0x7f9e4010d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 --2- 172.21.15.39:0/1296376422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e401049d0 0x7f9e40104de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 -- 172.21.15.39:0/1296376422 >> 172.21.15.39:0/1296376422 conn(0x7f9e40079740 msgr2=0x7f9e40079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 -- 172.21.15.39:0/1296376422 shutdown_connections 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.816+0000 7f9e46400640 1 -- 172.21.15.39:0/1296376422 wait complete. 2024-09-21T01:16:07.818 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:07 smithi039 ceph-mon[30415]: pgmap v780: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 250 op/s 2024-09-21T01:16:07.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.817+0000 7f9e46400640 1 Processor -- start 2024-09-21T01:16:07.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.817+0000 7f9e46400640 1 -- start start 2024-09-21T01:16:07.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e46400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e401049d0 0x7f9e4019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:07.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e46400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e40105320 0x7f9e4019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:07.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e46400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 0x7f9e401a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:07.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e46400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9e40113a90 con 0x7f9e401049d0 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e46400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9e40113910 con 0x7f9e4010de30 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e46400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9e40113c10 con 0x7f9e40105320 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e3fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 0x7f9e401a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e3ea00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e40105320 0x7f9e4019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e3fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 0x7f9e401a1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:56478/0 (socket says 172.21.15.39:56478) 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e3fe00640 1 -- 172.21.15.39:0/3264483780 learned_addr learned my addr 172.21.15.39:0/3264483780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:07.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.818+0000 7f9e3f400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e401049d0 0x7f9e4019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e3fe00640 1 -- 172.21.15.39:0/3264483780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e40105320 msgr2=0x7f9e4019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e3fe00640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e40105320 0x7f9e4019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e3fe00640 1 -- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e401049d0 msgr2=0x7f9e4019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e3fe00640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e401049d0 0x7f9e4019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e3fe00640 1 -- 172.21.15.39:0/3264483780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e401a2690 con 0x7f9e4010de30 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e3fe00640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 0x7f9e401a1f60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009fd0 tx=0x7f9e3800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38017070 con 0x7f9e4010de30 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e38003e20 con 0x7f9e4010de30 2024-09-21T01:16:07.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.819+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e401a2980 con 0x7f9e4010de30 2024-09-21T01:16:07.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.820+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38010460 con 0x7f9e4010de30 2024-09-21T01:16:07.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.820+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e401a90e0 con 0x7f9e4010de30 2024-09-21T01:16:07.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.821+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e4006c240 con 0x7f9e4010de30 2024-09-21T01:16:07.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.823+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f9e38003970 con 0x7f9e4010de30 2024-09-21T01:16:07.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.825+0000 7f9e37e00640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9e240b6700 0x7f9e240b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:07.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.825+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9599+0+0 (secure 0 0 0) 0x7f9e380e77c0 con 0x7f9e4010de30 2024-09-21T01:16:07.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.825+0000 7f9e3f400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9e240b6700 0x7f9e240b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:07.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.826+0000 7f9e3f400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9e240b6700 0x7f9e240b8c10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000c00 tx=0x7f9e30031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:07.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:07.829+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9e380b1c60 con 0x7f9e4010de30 2024-09-21T01:16:07.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:07 smithi160 ceph-mon[34696]: pgmap v780: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 250 op/s 2024-09-21T01:16:08.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.132+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e401a9340 con 0x7f9e4010de30 2024-09-21T01:16:08.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.133+0000 7f9e37e00640 1 -- 172.21.15.39:0/3264483780 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11918 (secure 0 0 0) 0x7f9e380b6ba0 con 0x7f9e4010de30 2024-09-21T01:16:08.135 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:16:08.135 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 40 2024-09-21T01:16:08.135 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"btime":"2024-09-21T01:15:30:474236+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:15:29.524010+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:16:08.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9e240b6700 msgr2=0x7f9e240b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9e240b6700 0x7f9e240b8c10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000c00 tx=0x7f9e30031400 comp rx=0 tx=0).stop 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 msgr2=0x7f9e401a1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 0x7f9e401a1f60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009fd0 tx=0x7f9e3800b260 comp rx=0 tx=0).stop 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 shutdown_connections 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9e240b6700 0x7f9e240b8c10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4010de30 0x7f9e401a1f60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e40105320 0x7f9e4019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 --2- 172.21.15.39:0/3264483780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e401049d0 0x7f9e4019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.136+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 >> 172.21.15.39:0/3264483780 conn(0x7f9e40079740 msgr2=0x7f9e4010cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.137+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 shutdown_connections 2024-09-21T01:16:08.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:08.137+0000 7f9e46400640 1 -- 172.21.15.39:0/3264483780 wait complete. 2024-09-21T01:16:08.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 518102 3.99 MB/sec execute 515 sec latency 58.497 ms 2024-09-21T01:16:08.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:08 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3264483780' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:08.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:08 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3264483780' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:08.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:08 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3264483780' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:09.141 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0' 2024-09-21T01:16:09.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0 2024-09-21T01:16:09.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 519220 4.00 MB/sec execute 516 sec latency 24.060 ms 2024-09-21T01:16:09.562 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:09.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:09 smithi150 ceph-mon[34945]: pgmap v781: 129 pgs: 129 active+clean; 420 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 1.6 MiB/s wr, 308 op/s 2024-09-21T01:16:09.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:09 smithi160 ceph-mon[34696]: pgmap v781: 129 pgs: 129 active+clean; 420 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 1.6 MiB/s wr, 308 op/s 2024-09-21T01:16:09.866 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:09 smithi039 ceph-mon[30415]: pgmap v781: 129 pgs: 129 active+clean; 420 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 1.6 MiB/s wr, 308 op/s 2024-09-21T01:16:10.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 520332 4.00 MB/sec execute 517 sec latency 20.218 ms 2024-09-21T01:16:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 -- 172.21.15.39:0/1776537303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109300 msgr2=0x7fdd38109710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1776537303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109300 0x7fdd38109710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009900 tx=0x7fdd2402a400 comp rx=0 tx=0).stop 2024-09-21T01:16:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 -- 172.21.15.39:0/1776537303 shutdown_connections 2024-09-21T01:16:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1776537303 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd38109c50 0x7fdd38110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1776537303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109300 0x7fdd38109710 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1776537303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd381039c0 0x7fdd38103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 -- 172.21.15.39:0/1776537303 >> 172.21.15.39:0/1776537303 conn(0x7fdd380ff430 msgr2=0x7fdd38101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 -- 172.21.15.39:0/1776537303 shutdown_connections 2024-09-21T01:16:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.401+0000 7fdd3e400640 1 -- 172.21.15.39:0/1776537303 wait complete. 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.402+0000 7fdd3e400640 1 Processor -- start 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 -- start start 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 0x7fdd3819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd38109300 0x7fdd3819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109c50 0x7fdd381a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38113910 con 0x7fdd38109300 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38113790 con 0x7fdd38109c50 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3e400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38113a90 con 0x7fdd381039c0 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 0x7fdd3819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 0x7fdd3819c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52858/0 (socket says 172.21.15.39:52858) 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.403+0000 7fdd3d000640 1 -- 172.21.15.39:0/1032239768 learned_addr learned my addr 172.21.15.39:0/1032239768 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd37e00640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd38109300 0x7fdd3819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109c50 0x7fdd381a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3d000640 1 -- 172.21.15.39:0/1032239768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109c50 msgr2=0x7fdd381a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3d000640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109c50 0x7fdd381a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3d000640 1 -- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd38109300 msgr2=0x7fdd3819cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3d000640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd38109300 0x7fdd3819cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3d000640 1 -- 172.21.15.39:0/1032239768 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd381a24f0 con 0x7fdd381039c0 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.404+0000 7fdd3d000640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 0x7fdd3819c580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00e9f0 tx=0x7fdd2c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.405+0000 7fdd35600640 1 -- 172.21.15.39:0/1032239768 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c00cd80 con 0x7fdd381039c0 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.405+0000 7fdd35600640 1 -- 172.21.15.39:0/1032239768 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd2c0108e0 con 0x7fdd381039c0 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.405+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd381a27e0 con 0x7fdd381039c0 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.405+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd381a8f40 con 0x7fdd381039c0 2024-09-21T01:16:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.405+0000 7fdd35600640 1 -- 172.21.15.39:0/1032239768 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c018980 con 0x7fdd381039c0 2024-09-21T01:16:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.409+0000 7fdd4661f740 1 Processor -- start 2024-09-21T01:16:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.409+0000 7fdd4661f740 1 -- start start 2024-09-21T01:16:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.409+0000 7fdd35600640 1 -- 172.21.15.39:0/1032239768 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fdd2c018b20 con 0x7fdd381039c0 2024-09-21T01:16:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291d99510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919c8980 0x55c291d99a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919d0b60 0x55c291d99f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd4661f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c291d0eca0 con 0x55c2919c8980 2024-09-21T01:16:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd4661f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c291cfe0d0 con 0x55c2919d0b60 2024-09-21T01:16:10.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd4661f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c291d01ee0 con 0x55c2919f1840 2024-09-21T01:16:10.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd3da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291d99510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd3da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291d99510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52866/0 (socket says 172.21.15.39:52866) 2024-09-21T01:16:10.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd3da00640 1 -- 172.21.15.39:0/641300249 learned_addr learned my addr 172.21.15.39:0/641300249 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:10.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd3d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919c8980 0x55c291d99a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd37e00640 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919d0b60 0x55c291d99f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd3da00640 1 -- 172.21.15.39:0/641300249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919d0b60 msgr2=0x55c291d99f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.410+0000 7fdd3da00640 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919d0b60 0x55c291d99f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd3da00640 1 -- 172.21.15.39:0/641300249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919c8980 msgr2=0x55c291d99a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd3da00640 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919c8980 0x55c291d99a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd3da00640 1 -- 172.21.15.39:0/641300249 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c291cb0400 con 0x55c2919f1840 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd3d000640 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919c8980 0x55c291d99a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd3da00640 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291d99510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009f50 tx=0x7fdd300118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857d39b4aabc6025 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd2a000640 1 -- 172.21.15.39:0/641300249 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004d20 con 0x55c2919f1840 2024-09-21T01:16:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.411+0000 7fdd35600640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd200b66f0 0x7fdd200b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.412+0000 7fdd2a000640 1 -- 172.21.15.39:0/641300249 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd30004050 con 0x55c2919f1840 2024-09-21T01:16:10.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.412+0000 7fdd2a000640 1 -- 172.21.15.39:0/641300249 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30005280 con 0x55c2919f1840 2024-09-21T01:16:10.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.412+0000 7fdd35600640 1 -- 172.21.15.39:0/1032239768 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9599+0+0 (secure 0 0 0) 0x7fdd2c0e7b90 con 0x7fdd381039c0 2024-09-21T01:16:10.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.412+0000 7fdd3d000640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd200b66f0 0x7fdd200b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 -- 172.21.15.39:0/641300249 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 msgr2=0x55c291d99510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291d99510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009f50 tx=0x7fdd300118d0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 -- 172.21.15.39:0/641300249 shutdown_connections 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919d0b60 0x55c291d99f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919c8980 0x55c291d99a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 --2- 172.21.15.39:0/641300249 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291d99510 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.413+0000 7fdd4661f740 1 -- 172.21.15.39:0/641300249 >> 172.21.15.39:0/641300249 conn(0x55c2919e5990 msgr2=0x55c291d97400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:10.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.414+0000 7fdd4661f740 1 -- 172.21.15.39:0/641300249 shutdown_connections 2024-09-21T01:16:10.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.414+0000 7fdd4661f740 1 -- 172.21.15.39:0/641300249 wait complete. 2024-09-21T01:16:10.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.414+0000 7fdd3d000640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd200b66f0 0x7fdd200b8c00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0e4f60 tx=0x7fdd2c000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:10.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.414+0000 7fdd4661f740 1 -- start start 2024-09-21T01:16:10.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 Processor -- start 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291da4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 0x55c291da5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919f1840 0x55c291db9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c291b5f7a0 con 0x55c2919f1840 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c291ca3a10 con 0x55c2919c8980 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55c291db0500 con 0x55c2919d0b60 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd4661f740 10 client.0 _ll_register_callbacks cb 0x55c291cb3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd37e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 0x55c291da5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.415+0000 7fdd3da00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291da4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 0x55c291da5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52876/0 (socket says 172.21.15.39:52876) 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 -- 172.21.15.39:0/465061427 learned_addr learned my addr 172.21.15.39:0/465061427 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd3da00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919f1840 0x55c291db9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 -- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 msgr2=0x55c291da4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291da4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 -- 172.21.15.39:0/465061427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919f1840 msgr2=0x55c291db9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919f1840 0x55c291db9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 -- 172.21.15.39:0/465061427 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c291a6bb90 con 0x55c2919d0b60 2024-09-21T01:16:10.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd3da00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919f1840 0x55c291db9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:10.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd37e00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 0x55c291da5110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806b920 tx=0x7fdd24032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9fb49dbb9726e17 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:10.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.416+0000 7fdd1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:16:10.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd1ea00640 1 -- 172.21.15.39:0/465061427 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24035040 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd1ea00640 1 -- 172.21.15.39:0/465061427 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9599+0+0 (secure 0 0 0) 0x7fdd240316a0 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd30004050 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd1ea00640 1 -- 172.21.15.39:0/465061427 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdd2404e050 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd4661f740 10 client.26029 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c291dbc240 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.417+0000 7fdd1ea00640 1 -- 172.21.15.39:0/465061427 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd30004050 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd1ea00640 1 -- 172.21.15.39:0/465061427 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fdd240054c0 con 0x55c2919d0b60 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd4661f740 10 client.26029 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd4661f740 10 client.26029 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd4661f740 20 client.26029 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd4661f740 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c2919b01c0 0x55c291dbf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd4661f740 4 client.26029 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c291d10bf0 con 0x55c2919b01c0 2024-09-21T01:16:10.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.418+0000 7fdd3da00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c2919b01c0 0x55c291dbf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.419+0000 7fdd3da00640 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c2919b01c0 0x55c291dbf7e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48652688aff9200 server_cookie=a17224db93ddacbe in_seq=0 out_seq=0 2024-09-21T01:16:10.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.419+0000 7fdd1ea00640 10 client.26029 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:10.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.420+0000 7fdd1ea00640 1 -- 172.21.15.39:0/465061427 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c291d10bf0 con 0x55c2919b01c0 2024-09-21T01:16:10.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.420+0000 7fdd1ea00640 10 client.26029 handle_command_reply: tid=0 2024-09-21T01:16:10.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.420+0000 7fdd4661f740 1 client.26029 shutdown 2024-09-21T01:16:10.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.420+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 msgr2=0x55c291da5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.420+0000 7fdd4661f740 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 0x55c291da5110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806b920 tx=0x7fdd24032da0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 shutdown_connections 2024-09-21T01:16:10.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c2919b01c0 0x55c291dbf7e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919f1840 0x55c291db9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919d0b60 0x55c291da5110 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 --2- 172.21.15.39:0/465061427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291da4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd1ea00640 0 client.26029 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:10.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 >> 172.21.15.39:0/465061427 conn(0x55c2919e5990 msgr2=0x55c291d9c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:10.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 shutdown_connections 2024-09-21T01:16:10.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 1 -- 172.21.15.39:0/465061427 wait complete. 2024-09-21T01:16:10.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.422+0000 7fdd4661f740 20 client.26029 trim_cache size 0 max 16384 2024-09-21T01:16:10.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 Processor -- start 2024-09-21T01:16:10.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 -- start start 2024-09-21T01:16:10.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 0x55c291d9af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919f1840 0x55c291d9ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd18012ad0 con 0x55c2919d0b60 2024-09-21T01:16:10.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd18010310 con 0x55c2919f1840 2024-09-21T01:16:10.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.444+0000 7fdd4661f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1800e180 con 0x55c2919c8980 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 0x55c291d9af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 0x55c291d9af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52890/0 (socket says 172.21.15.39:52890) 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 -- 172.21.15.39:0/1059026803 learned_addr learned my addr 172.21.15.39:0/1059026803 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919f1840 0x55c291d9ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd37e00640 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 -- 172.21.15.39:0/1059026803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919f1840 msgr2=0x55c291d9ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919f1840 0x55c291d9ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 -- 172.21.15.39:0/1059026803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 msgr2=0x55c291db9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.445+0000 7fdd3da00640 1 -- 172.21.15.39:0/1059026803 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c291da6fa0 con 0x55c2919c8980 2024-09-21T01:16:10.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.446+0000 7fdd37e00640 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:10.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.446+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 0x55c291d9af00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdd3001b040 tx=0x7fdd30009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60717e5c747945c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:10.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.446+0000 7fdd29600640 1 -- 172.21.15.39:0/1059026803 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004350 con 0x55c2919c8980 2024-09-21T01:16:10.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.446+0000 7fdd29600640 1 -- 172.21.15.39:0/1059026803 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd300044f0 con 0x55c2919c8980 2024-09-21T01:16:10.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.446+0000 7fdd29600640 1 -- 172.21.15.39:0/1059026803 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30005630 con 0x55c2919c8980 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.447+0000 7fdd4661f740 1 -- 172.21.15.39:0/1059026803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 msgr2=0x55c291d9af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.447+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 0x55c291d9af00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdd3001b040 tx=0x7fdd30009500 comp rx=0 tx=0).stop 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 -- 172.21.15.39:0/1059026803 shutdown_connections 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919f1840 0x55c291d9ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1059026803 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919c8980 0x55c291d9af00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 -- 172.21.15.39:0/1059026803 >> 172.21.15.39:0/1059026803 conn(0x55c2919e5990 msgr2=0x55c291d9e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:10.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 -- 172.21.15.39:0/1059026803 shutdown_connections 2024-09-21T01:16:10.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.448+0000 7fdd4661f740 1 -- 172.21.15.39:0/1059026803 wait complete. 2024-09-21T01:16:10.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 -- start start 2024-09-21T01:16:10.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 Processor -- start 2024-09-21T01:16:10.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291db3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291e291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1800e7d0 con 0x55c2919d0b60 2024-09-21T01:16:10.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.449+0000 7fdd4661f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1800fc40 con 0x55c2919c8980 2024-09-21T01:16:10.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd4661f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1800eea0 con 0x55c2919f1840 2024-09-21T01:16:10.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd4661f740 10 client.0 _ll_register_callbacks cb 0x7fdd300093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47536/0 (socket says 172.21.15.39:47536) 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 -- 172.21.15.39:0/1918869157 learned_addr learned my addr 172.21.15.39:0/1918869157 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291db3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291e291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 -- 172.21.15.39:0/1918869157 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 msgr2=0x55c291e291f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291e291f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 -- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 msgr2=0x55c291db3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:10.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291db3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd37e00640 1 -- 172.21.15.39:0/1918869157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c291e29760 con 0x55c2919d0b60 2024-09-21T01:16:10.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.450+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291db3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:10.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd37e00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db4450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806b920 tx=0x7fdd24009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bcd1a3d3dce1e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:10.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd1ea00640 1 -- 172.21.15.39:0/1918869157 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24057060 con 0x55c2919d0b60 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd1ea00640 1 -- 172.21.15.39:0/1918869157 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9599+0+0 (secure 0 0 0) 0x7fdd240316a0 con 0x55c2919d0b60 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd300044f0 con 0x55c2919d0b60 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd1ea00640 1 -- 172.21.15.39:0/1918869157 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdd24035070 con 0x55c2919d0b60 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.451+0000 7fdd4661f740 10 client.16014 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.452+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c291e2a9c0 con 0x55c2919d0b60 2024-09-21T01:16:10.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.452+0000 7fdd1ea00640 1 -- 172.21.15.39:0/1918869157 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd300044f0 con 0x55c2919d0b60 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd1ea00640 1 -- 172.21.15.39:0/1918869157 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fdd240318a0 con 0x55c2919d0b60 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd4661f740 10 client.16014 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd4661f740 10 client.16014 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd4661f740 20 client.16014 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c291985570 0x55c291e2d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd4661f740 4 client.16014 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}] 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}) -- 0x55c291d0c6a0 con 0x55c291985570 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.453+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c291985570 0x55c291e2d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:10.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.454+0000 7fdd1ea00640 10 client.16014 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:10.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.454+0000 7fdd3da00640 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c291985570 0x55c291e2d1e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f889f5de70c835a server_cookie=571d56bab1de2303 in_seq=0 out_seq=0 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd1ea00640 1 -- 172.21.15.39:0/1918869157 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c291d0c6a0 con 0x55c291985570 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd1ea00640 10 client.16014 handle_command_reply: tid=0 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 client.16014 shutdown 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 msgr2=0x55c291db4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db4450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806b920 tx=0x7fdd24009800 comp rx=0 tx=0).stop 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 shutdown_connections 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55c291985570 0x55c291e2d1e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55c2919f1840 0x55c291e291f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c2919d0b60 0x55c291db4450 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 --2- 172.21.15.39:0/1918869157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2919c8980 0x55c291db3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 >> 172.21.15.39:0/1918869157 conn(0x55c2919e5990 msgr2=0x55c291d9acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd1ea00640 0 client.16014 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 shutdown_connections 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 1 -- 172.21.15.39:0/1918869157 wait complete. 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.646+0000 7fdd4661f740 20 client.16014 trim_cache size 0 max 16384 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 40, 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 46, 2024-09-21T01:16:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "c746d144": { 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 46, 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 137.8, 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd200b66f0 msgr2=0x7fdd200b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd200b66f0 0x7fdd200b8c00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0e4f60 tx=0x7fdd2c000bc0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 msgr2=0x7fdd3819c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 0x7fdd3819c580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00e9f0 tx=0x7fdd2c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 shutdown_connections 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdd200b66f0 0x7fdd200b8c00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38109c50 0x7fdd381a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd38109300 0x7fdd3819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 --2- 172.21.15.39:0/1032239768 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdd381039c0 0x7fdd3819c580 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:10.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 >> 172.21.15.39:0/1032239768 conn(0x7fdd380ff430 msgr2=0x7fdd3810e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:10.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.647+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 shutdown_connections 2024-09-21T01:16:10.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:10.648+0000 7fdd3e400640 1 -- 172.21.15.39:0/1032239768 wait complete. 2024-09-21T01:16:10.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:10 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:16:10.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:10 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:16:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:10 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:16:11.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.99 MB/sec execute 518 sec latency 716.284 ms 2024-09-21T01:16:11.486 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c746d144', quiesce took 4.9 seconds. Will release after: 12.9 2024-09-21T01:16:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:11 smithi160 ceph-mon[34696]: pgmap v782: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 1.3 MiB/s wr, 276 op/s 2024-09-21T01:16:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:16:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:16:11.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:16:11.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:11 smithi039 ceph-mon[30415]: pgmap v782: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 1.3 MiB/s wr, 276 op/s 2024-09-21T01:16:11.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:16:11.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:16:11.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:16:12.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:11 smithi150 ceph-mon[34945]: pgmap v782: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 1.3 MiB/s wr, 276 op/s 2024-09-21T01:16:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:16:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:16:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:16:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.99 MB/sec execute 519 sec latency 1716.419 ms 2024-09-21T01:16:13.196 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.98 MB/sec execute 520 sec latency 2716.520 ms 2024-09-21T01:16:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:13 smithi160 ceph-mon[34696]: pgmap v783: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-09-21T01:16:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:13 smithi039 ceph-mon[30415]: pgmap v783: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-09-21T01:16:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:13 smithi150 ceph-mon[34945]: pgmap v783: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-09-21T01:16:14.197 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.97 MB/sec execute 521 sec latency 3716.636 ms 2024-09-21T01:16:15.197 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.96 MB/sec execute 522 sec latency 4716.760 ms 2024-09-21T01:16:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:15 smithi150 ceph-mon[34945]: pgmap v784: 129 pgs: 129 active+clean; 426 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 1.6 MiB/s wr, 218 op/s 2024-09-21T01:16:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:15 smithi150 ceph-mon[34945]: osdmap e104: 12 total, 12 up, 12 in 2024-09-21T01:16:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:15 smithi160 ceph-mon[34696]: pgmap v784: 129 pgs: 129 active+clean; 426 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 1.6 MiB/s wr, 218 op/s 2024-09-21T01:16:15.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:15 smithi160 ceph-mon[34696]: osdmap e104: 12 total, 12 up, 12 in 2024-09-21T01:16:15.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:15 smithi039 ceph-mon[30415]: pgmap v784: 129 pgs: 129 active+clean; 426 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 1.6 MiB/s wr, 218 op/s 2024-09-21T01:16:15.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:15 smithi039 ceph-mon[30415]: osdmap e104: 12 total, 12 up, 12 in 2024-09-21T01:16:16.210 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.96 MB/sec execute 523 sec latency 5716.936 ms 2024-09-21T01:16:17.197 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.95 MB/sec execute 524 sec latency 6717.077 ms 2024-09-21T01:16:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:17 smithi150 ceph-mon[34945]: pgmap v786: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-09-21T01:16:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:17 smithi160 ceph-mon[34696]: pgmap v786: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-09-21T01:16:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:17 smithi039 ceph-mon[30415]: pgmap v786: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-09-21T01:16:18.198 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.94 MB/sec execute 525 sec latency 7718.412 ms 2024-09-21T01:16:19.198 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.93 MB/sec execute 526 sec latency 8718.475 ms 2024-09-21T01:16:19.550 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:19 smithi039 ceph-mon[30415]: pgmap v787: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 843 KiB/s wr, 85 op/s 2024-09-21T01:16:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:19 smithi150 ceph-mon[34945]: pgmap v787: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 843 KiB/s wr, 85 op/s 2024-09-21T01:16:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:19 smithi160 ceph-mon[34696]: pgmap v787: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 843 KiB/s wr, 85 op/s 2024-09-21T01:16:20.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.92 MB/sec execute 527 sec latency 9718.526 ms 2024-09-21T01:16:21.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.92 MB/sec execute 528 sec latency 10718.574 ms 2024-09-21T01:16:21.667 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:21 smithi039 ceph-mon[30415]: pgmap v788: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 79 op/s 2024-09-21T01:16:21.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:21 smithi150 ceph-mon[34945]: pgmap v788: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 79 op/s 2024-09-21T01:16:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:21 smithi160 ceph-mon[34696]: pgmap v788: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 79 op/s 2024-09-21T01:16:22.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.91 MB/sec execute 529 sec latency 11718.632 ms 2024-09-21T01:16:23.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.90 MB/sec execute 530 sec latency 12718.726 ms 2024-09-21T01:16:23.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:23 smithi150 ceph-mon[34945]: pgmap v789: 129 pgs: 129 active+clean; 436 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-21T01:16:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:23 smithi160 ceph-mon[34696]: pgmap v789: 129 pgs: 129 active+clean; 436 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-21T01:16:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:23 smithi039 ceph-mon[30415]: pgmap v789: 129 pgs: 129 active+clean; 436 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-21T01:16:24.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.90 MB/sec execute 531 sec latency 13718.782 ms 2024-09-21T01:16:24.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:16:24.782 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:25.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.89 MB/sec execute 532 sec latency 14718.895 ms 2024-09-21T01:16:25.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 -- 172.21.15.39:0/90083906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca41039b0 msgr2=0x7fdca4103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:25.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 --2- 172.21.15.39:0/90083906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca41039b0 0x7fdca4103dc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009900 tx=0x7fdc9402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 -- 172.21.15.39:0/90083906 shutdown_connections 2024-09-21T01:16:25.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 --2- 172.21.15.39:0/90083906 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca4109c40 0x7fdca4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 --2- 172.21.15.39:0/90083906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41092f0 0x7fdca4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 --2- 172.21.15.39:0/90083906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca41039b0 0x7fdca4103dc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.581+0000 7fdca3e00640 1 -- 172.21.15.39:0/90083906 >> 172.21.15.39:0/90083906 conn(0x7fdca40ff400 msgr2=0x7fdca4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:25.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.582+0000 7fdca3e00640 1 -- 172.21.15.39:0/90083906 shutdown_connections 2024-09-21T01:16:25.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.582+0000 7fdca3e00640 1 -- 172.21.15.39:0/90083906 wait complete. 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 Processor -- start 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 -- start start 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 0x7fdca419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41092f0 0x7fdca419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 0x7fdca41a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdca4113900 con 0x7fdca4109c40 2024-09-21T01:16:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdca4113780 con 0x7fdca41092f0 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.583+0000 7fdca3e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fdca4113a80 con 0x7fdca41039b0 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 0x7fdca419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 0x7fdca41a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 0x7fdca419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36410/0 (socket says 172.21.15.39:36410) 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 -- 172.21.15.39:0/85234658 learned_addr learned my addr 172.21.15.39:0/85234658 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 0x7fdca41a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48976/0 (socket says 172.21.15.39:48976) 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41092f0 0x7fdca419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 -- 172.21.15.39:0/85234658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41092f0 msgr2=0x7fdca419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41092f0 0x7fdca419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 -- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 msgr2=0x7fdca41a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 0x7fdca41a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 -- 172.21.15.39:0/85234658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca41a2460 con 0x7fdca41039b0 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca3400640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 0x7fdca41a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:25.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.584+0000 7fdca2a00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 0x7fdca419c4f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc94005940 tx=0x7fdc940040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:25.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.585+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9403f030 con 0x7fdca41039b0 2024-09-21T01:16:25.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.585+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc940044c0 con 0x7fdca41039b0 2024-09-21T01:16:25.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.585+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca41a26f0 con 0x7fdca41039b0 2024-09-21T01:16:25.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.585+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94035070 con 0x7fdca41039b0 2024-09-21T01:16:25.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.585+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca41a8eb0 con 0x7fdca41039b0 2024-09-21T01:16:25.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.586+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca4104c30 con 0x7fdca41039b0 2024-09-21T01:16:25.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.588+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fdc94042030 con 0x7fdca41039b0 2024-09-21T01:16:25.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.589+0000 7fdca8a00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdc880b6700 0x7fdc880b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:25.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.590+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fdc94107ad0 con 0x7fdca41039b0 2024-09-21T01:16:25.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.590+0000 7fdca2000640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdc880b6700 0x7fdc880b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:25.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.591+0000 7fdca2000640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdc880b6700 0x7fdc880b8c10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdca4105ab0 tx=0x7fdc90005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:25.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.594+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdc940d1fc0 con 0x7fdca41039b0 2024-09-21T01:16:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:25 smithi150 ceph-mon[34945]: pgmap v790: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 2.6 MiB/s wr, 85 op/s 2024-09-21T01:16:25.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.831+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdca41a9080 con 0x7fdca41039b0 2024-09-21T01:16:25.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:25 smithi039 ceph-mon[30415]: pgmap v790: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 2.6 MiB/s wr, 85 op/s 2024-09-21T01:16:25.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.832+0000 7fdca8a00640 1 -- 172.21.15.39:0/85234658 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11918 (secure 0 0 0) 0x7fdc940d6f00 con 0x7fdca41039b0 2024-09-21T01:16:25.833 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:16:25.833 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"btime":"2024-09-21T01:15:30:474236+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:15:29.524010+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:16:25.833 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 40 2024-09-21T01:16:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdc880b6700 msgr2=0x7fdc880b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdc880b6700 0x7fdc880b8c10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdca4105ab0 tx=0x7fdc90005f90 comp rx=0 tx=0).stop 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 msgr2=0x7fdca419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 0x7fdca419c4f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc94005940 tx=0x7fdc940040c0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 shutdown_connections 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca4109c40 0x7fdca41a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fdc880b6700 0x7fdc880b8c10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41092f0 0x7fdca419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 --2- 172.21.15.39:0/85234658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdca41039b0 0x7fdca419c4f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:25.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 >> 172.21.15.39:0/85234658 conn(0x7fdca40ff400 msgr2=0x7fdca41089e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:25.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 shutdown_connections 2024-09-21T01:16:25.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:25.835+0000 7fdca3e00640 1 -- 172.21.15.39:0/85234658 wait complete. 2024-09-21T01:16:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:25 smithi160 ceph-mon[34696]: pgmap v790: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 2.6 MiB/s wr, 85 op/s 2024-09-21T01:16:26.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.88 MB/sec execute 533 sec latency 15719.022 ms 2024-09-21T01:16:26.758 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:26 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/85234658' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:26.796 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id c746d144 --release --await' 2024-09-21T01:16:26.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id c746d144 --release --await 2024-09-21T01:16:26.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:26 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/85234658' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:26.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:26 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/85234658' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:27.182 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:27.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521063 3.87 MB/sec execute 534 sec latency 16719.084 ms 2024-09-21T01:16:27.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:27 smithi039 ceph-mon[30415]: pgmap v791: 129 pgs: 129 active+clean; 435 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 2.2 MiB/s wr, 77 op/s 2024-09-21T01:16:27.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:27 smithi150 ceph-mon[34945]: pgmap v791: 129 pgs: 129 active+clean; 435 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 2.2 MiB/s wr, 77 op/s 2024-09-21T01:16:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:27 smithi160 ceph-mon[34696]: pgmap v791: 129 pgs: 129 active+clean; 435 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 2.2 MiB/s wr, 77 op/s 2024-09-21T01:16:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.889+0000 7fc813000640 1 -- 172.21.15.39:0/2351473008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c10be40 msgr2=0x7fc80c112670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.889+0000 7fc813000640 1 --2- 172.21.15.39:0/2351473008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c10be40 0x7fc80c112670 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc804009ca0 tx=0x7fc80402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 -- 172.21.15.39:0/2351473008 shutdown_connections 2024-09-21T01:16:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 --2- 172.21.15.39:0/2351473008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c10be40 0x7fc80c112670 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 --2- 172.21.15.39:0/2351473008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc80c10b4f0 0x7fc80c10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 --2- 172.21.15.39:0/2351473008 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c105bb0 0x7fc80c105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 -- 172.21.15.39:0/2351473008 >> 172.21.15.39:0/2351473008 conn(0x7fc80c101680 msgr2=0x7fc80c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:27.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 -- 172.21.15.39:0/2351473008 shutdown_connections 2024-09-21T01:16:27.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.890+0000 7fc813000640 1 -- 172.21.15.39:0/2351473008 wait complete. 2024-09-21T01:16:27.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.891+0000 7fc813000640 1 Processor -- start 2024-09-21T01:16:27.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 -- start start 2024-09-21T01:16:27.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c105bb0 0x7fc80c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 0x7fc80c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc80c10be40 0x7fc80c1a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c115b00 con 0x7fc80c105bb0 2024-09-21T01:16:27.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c115980 con 0x7fc80c10be40 2024-09-21T01:16:27.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc813000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c115c80 con 0x7fc80c10b4f0 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc811200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 0x7fc80c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc811c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c105bb0 0x7fc80c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc811200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 0x7fc80c19ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36422/0 (socket says 172.21.15.39:36422) 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc811200640 1 -- 172.21.15.39:0/3714367395 learned_addr learned my addr 172.21.15.39:0/3714367395 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.892+0000 7fc812600640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc80c10be40 0x7fc80c1a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc811200640 1 -- 172.21.15.39:0/3714367395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc80c10be40 msgr2=0x7fc80c1a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc811200640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc80c10be40 0x7fc80c1a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc811200640 1 -- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c105bb0 msgr2=0x7fc80c19c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc811200640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c105bb0 0x7fc80c19c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc811200640 1 -- 172.21.15.39:0/3714367395 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c1a2440 con 0x7fc80c10b4f0 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc811200640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 0x7fc80c19ca70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800e9f0 tx=0x7fc7f800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.893+0000 7fc80a000640 1 -- 172.21.15.39:0/3714367395 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f800ccc0 con 0x7fc80c10b4f0 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.894+0000 7fc80a000640 1 -- 172.21.15.39:0/3714367395 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f80108e0 con 0x7fc80c10b4f0 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.894+0000 7fc80a000640 1 -- 172.21.15.39:0/3714367395 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8018860 con 0x7fc80c10b4f0 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.894+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c1a2730 con 0x7fc80c10b4f0 2024-09-21T01:16:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.894+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c1a8e90 con 0x7fc80c10b4f0 2024-09-21T01:16:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 Processor -- start 2024-09-21T01:16:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 -- start start 2024-09-21T01:16:27.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df259340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df259880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df259dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5627df1d0290 con 0x5627deeb2520 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5627df1bf6c0 con 0x5627dee89660 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.897+0000 7fc81b27f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5627df1c34d0 con 0x5627dee91840 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df259340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df259340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49012/0 (socket says 172.21.15.39:49012) 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 -- 172.21.15.39:0/1605375896 learned_addr learned my addr 172.21.15.39:0/1605375896 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc811c00640 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df259880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc80a000640 1 -- 172.21.15.39:0/3714367395 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fc7f8010430 con 0x7fc80c10b4f0 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 -- 172.21.15.39:0/1605375896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 msgr2=0x5627df259dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df259dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 -- 172.21.15.39:0/1605375896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 msgr2=0x5627df259880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df259880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc812600640 1 -- 172.21.15.39:0/1605375896 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627df14c460 con 0x5627deeb2520 2024-09-21T01:16:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.898+0000 7fc811c00640 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df259880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.899+0000 7fc812600640 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df259340 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc804009ca0 tx=0x7fc804009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb6d3d9b14446b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.899+0000 7fc7fea00640 1 -- 172.21.15.39:0/1605375896 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804031db0 con 0x5627deeb2520 2024-09-21T01:16:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.899+0000 7fc7fea00640 1 -- 172.21.15.39:0/1605375896 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc804036030 con 0x5627deeb2520 2024-09-21T01:16:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.899+0000 7fc7fea00640 1 -- 172.21.15.39:0/1605375896 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804004260 con 0x5627deeb2520 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 -- 172.21.15.39:0/1605375896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 msgr2=0x5627df259340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df259340 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc804009ca0 tx=0x7fc804009800 comp rx=0 tx=0).stop 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 -- 172.21.15.39:0/1605375896 shutdown_connections 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df259dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df259880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1605375896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df259340 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 -- 172.21.15.39:0/1605375896 >> 172.21.15.39:0/1605375896 conn(0x5627deea6910 msgr2=0x5627df257230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 -- 172.21.15.39:0/1605375896 shutdown_connections 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc80a000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7f40b6630 0x7fc7f40b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc80a000640 1 -- 172.21.15.39:0/3714367395 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fc7f80e7d90 con 0x7fc80c10b4f0 2024-09-21T01:16:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.900+0000 7fc81b27f740 1 -- 172.21.15.39:0/1605375896 wait complete. 2024-09-21T01:16:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.901+0000 7fc811c00640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7f40b6630 0x7fc7f40b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.901+0000 7fc81b27f740 1 -- start start 2024-09-21T01:16:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.901+0000 7fc81b27f740 1 Processor -- start 2024-09-21T01:16:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.901+0000 7fc811c00640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7f40b6630 0x7fc7f40b8b40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5627df257fc0 tx=0x7fc80000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 0x5627df264f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df265460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df270c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5627df020bb0 con 0x5627dee89660 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5627df15a6c0 con 0x5627deeb2520 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5627df267940 con 0x5627dee91840 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc81b27f740 10 client.0 _ll_register_callbacks cb 0x5627df175490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc811c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 0x5627df264f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc811c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 0x5627df264f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49026/0 (socket says 172.21.15.39:49026) 2024-09-21T01:16:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc811c00640 1 -- 172.21.15.39:0/2423429562 learned_addr learned my addr 172.21.15.39:0/2423429562 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc812600640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df270c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.902+0000 7fc811200640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df265460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811c00640 1 -- 172.21.15.39:0/2423429562 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 msgr2=0x5627df265460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811c00640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df265460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811c00640 1 -- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 msgr2=0x5627df270c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811c00640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df270c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811c00640 1 -- 172.21.15.39:0/2423429562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627def1db00 con 0x5627dee89660 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811200640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df265460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc811c00640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 0x5627df264f20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc800010790 tx=0x7fc8000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92332a5e71b5743 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc7f3400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc7f3400640 1 -- 172.21.15.39:0/2423429562 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800005220 con 0x5627dee89660 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc7f3400640 1 -- 172.21.15.39:0/2423429562 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fc8000053c0 con 0x5627dee89660 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.903+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc804036030 con 0x5627dee89660 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc7f3400640 1 -- 172.21.15.39:0/2423429562 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc80002f050 con 0x5627dee89660 2024-09-21T01:16:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 10 client.16032 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5627def94960 con 0x5627dee89660 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc7f3400640 1 -- 172.21.15.39:0/2423429562 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc804036030 con 0x5627dee89660 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc7f3400640 1 -- 172.21.15.39:0/2423429562 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fc800004b30 con 0x5627dee89660 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 10 client.16032 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 10 client.16032 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 20 client.16032 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee70ea0 0x5627df276d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 4 client.16032 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.904+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627df1d21e0 con 0x5627dee70ea0 2024-09-21T01:16:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.905+0000 7fc811200640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee70ea0 0x5627df276d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.905+0000 7fc811200640 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee70ea0 0x5627df276d40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b856f4f9c9e36c server_cookie=311384d72c9f20c9 in_seq=0 out_seq=0 2024-09-21T01:16:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.905+0000 7fc7f3400640 10 client.16032 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc7f3400640 1 -- 172.21.15.39:0/2423429562 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5627df1d21e0 con 0x5627dee70ea0 2024-09-21T01:16:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc7f3400640 10 client.16032 handle_command_reply: tid=0 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc81b27f740 1 client.16032 shutdown 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 msgr2=0x5627df264f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc81b27f740 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 0x5627df264f20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc800010790 tx=0x7fc8000046c0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 shutdown_connections 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.906+0000 7fc81b27f740 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee70ea0 0x5627df276d40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df270c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df265460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 1 --2- 172.21.15.39:0/2423429562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee89660 0x5627df264f20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 >> 172.21.15.39:0/2423429562 conn(0x5627deea6910 msgr2=0x5627df25c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc7f3400640 0 client.16032 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 shutdown_connections 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 1 -- 172.21.15.39:0/2423429562 wait complete. 2024-09-21T01:16:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.907+0000 7fc81b27f740 20 client.16032 trim_cache size 0 max 16384 2024-09-21T01:16:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 Processor -- start 2024-09-21T01:16:27.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 -- start start 2024-09-21T01:16:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df270b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df271070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df25c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1b5980 con 0x5627deeb2520 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1b31c0 con 0x5627dee89660 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.925+0000 7fc81b27f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1b1030 con 0x5627dee91840 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df25c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df25c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49034/0 (socket says 172.21.15.39:49034) 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 -- 172.21.15.39:0/1980609359 learned_addr learned my addr 172.21.15.39:0/1980609359 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc811200640 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df271070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc811c00640 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df270b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 -- 172.21.15.39:0/1980609359 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 msgr2=0x5627df271070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df271070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 -- 172.21.15.39:0/1980609359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 msgr2=0x5627df270b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df270b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.926+0000 7fc812600640 1 -- 172.21.15.39:0/1980609359 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627df2655e0 con 0x5627deeb2520 2024-09-21T01:16:27.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.927+0000 7fc812600640 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df25c440 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc804009a60 tx=0x7fc804003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa4b047b9d3a00c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.927+0000 7fc7fe000640 1 -- 172.21.15.39:0/1980609359 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80404a040 con 0x5627deeb2520 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.927+0000 7fc7fe000640 1 -- 172.21.15.39:0/1980609359 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc804036040 con 0x5627deeb2520 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.927+0000 7fc7fe000640 1 -- 172.21.15.39:0/1980609359 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80402f8c0 con 0x5627deeb2520 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 -- 172.21.15.39:0/1980609359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 msgr2=0x5627df25c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df25c440 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc804009a60 tx=0x7fc804003d90 comp rx=0 tx=0).stop 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 -- 172.21.15.39:0/1980609359 shutdown_connections 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627deeb2520 0x5627df25c440 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee91840 0x5627df271070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 --2- 172.21.15.39:0/1980609359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627dee89660 0x5627df270b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 -- 172.21.15.39:0/1980609359 >> 172.21.15.39:0/1980609359 conn(0x5627deea6910 msgr2=0x5627df2635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 -- 172.21.15.39:0/1980609359 shutdown_connections 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 -- 172.21.15.39:0/1980609359 wait complete. 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.928+0000 7fc81b27f740 1 -- start start 2024-09-21T01:16:27.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 Processor -- start 2024-09-21T01:16:27.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 0x5627df27c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee91840 0x5627df27c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df27ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1b1680 con 0x5627dee91840 2024-09-21T01:16:27.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1b2af0 con 0x5627deeb2520 2024-09-21T01:16:27.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1b1d50 con 0x5627dee89660 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc81b27f740 10 client.0 _ll_register_callbacks cb 0x7fc80000b950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc811c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 0x5627df27c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc811200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee91840 0x5627df27c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc811c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 0x5627df27c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36456/0 (socket says 172.21.15.39:36456) 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.929+0000 7fc811c00640 1 -- 172.21.15.39:0/3254755003 learned_addr learned my addr 172.21.15.39:0/3254755003 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc812600640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df27ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811c00640 1 -- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 msgr2=0x5627df27ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811c00640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df27ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811c00640 1 -- 172.21.15.39:0/3254755003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee91840 msgr2=0x5627df27c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811c00640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee91840 0x5627df27c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811c00640 1 -- 172.21.15.39:0/3254755003 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627df26c290 con 0x5627dee89660 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811200640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee91840 0x5627df27c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:27.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc811c00640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 0x5627df27c410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc800010790 tx=0x7fc8000049e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eacd451ae9d71f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.930+0000 7fc7f3400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc7f3400640 1 -- 172.21.15.39:0/3254755003 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8000051d0 con 0x5627dee89660 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc804036040 con 0x5627dee89660 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc7f3400640 1 -- 172.21.15.39:0/3254755003 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fc800005370 con 0x5627dee89660 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc7f3400640 1 -- 172.21.15.39:0/3254755003 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc800047050 con 0x5627dee89660 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc81b27f740 10 client.26056 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5627df2e19a0 con 0x5627dee89660 2024-09-21T01:16:27.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc7f3400640 1 -- 172.21.15.39:0/3254755003 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc804036040 con 0x5627dee89660 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.931+0000 7fc7f3400640 1 -- 172.21.15.39:0/3254755003 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fc80003c020 con 0x5627dee89660 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc81b27f740 10 client.26056 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc81b27f740 10 client.26056 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc81b27f740 20 client.26056 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc81b27f740 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee46250 0x5627df2e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc81b27f740 4 client.26056 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c746d144", "release": true, "await": true}] 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c746d144", "release": true, "await": true}) -- 0x5627df1cdc90 con 0x5627dee46250 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.932+0000 7fc811200640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee46250 0x5627df2e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.933+0000 7fc811200640 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee46250 0x5627df2e40a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c356855fbd45c4f6 server_cookie=d249cae10c29303b in_seq=0 out_seq=0 2024-09-21T01:16:27.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.933+0000 7fc7f3400640 10 client.26056 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:27.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.939+0000 7fc7f3400640 1 -- 172.21.15.39:0/3254755003 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5627df1cdc90 con 0x5627dee46250 2024-09-21T01:16:27.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.939+0000 7fc7f3400640 10 client.26056 handle_command_reply: tid=0 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.939+0000 7fc81b27f740 1 client.26056 shutdown 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.939+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 msgr2=0x5627df27c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.939+0000 7fc81b27f740 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 0x5627df27c410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc800010790 tx=0x7fc8000049e0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 shutdown_connections 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5627dee46250 0x5627df2e40a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5627deeb2520 0x5627df27ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5627dee91840 0x5627df27c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 --2- 172.21.15.39:0/3254755003 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5627dee89660 0x5627df27c410 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc7f3400640 0 client.26056 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 >> 172.21.15.39:0/3254755003 conn(0x5627deea6910 msgr2=0x5627df25c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 shutdown_connections 2024-09-21T01:16:27.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 1 -- 172.21.15.39:0/3254755003 wait complete. 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.940+0000 7fc81b27f740 20 client.26056 trim_cache size 0 max 16384 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 40, 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 48, 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:16:27.942 INFO:teuthology.orchestra.run.smithi039.stdout: "c746d144": { 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 48, 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 137.8, 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:16:27.943 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:27.944 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:16:27.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7f40b6630 msgr2=0x7fc7f40b8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7f40b6630 0x7fc7f40b8b40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5627df257fc0 tx=0x7fc80000b410 comp rx=0 tx=0).stop 2024-09-21T01:16:27.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 msgr2=0x7fc80c19ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 0x7fc80c19ca70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800e9f0 tx=0x7fc7f800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 shutdown_connections 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc7f40b6630 0x7fc7f40b8b40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc80c10be40 0x7fc80c1a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc80c10b4f0 0x7fc80c19ca70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 --2- 172.21.15.39:0/3714367395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc80c105bb0 0x7fc80c19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 >> 172.21.15.39:0/3714367395 conn(0x7fc80c101680 msgr2=0x7fc80c10ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 shutdown_connections 2024-09-21T01:16:27.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:27.941+0000 7fc813000640 1 -- 172.21.15.39:0/3714367395 wait complete. 2024-09-21T01:16:28.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 521587 3.88 MB/sec execute 535 sec latency 17458.947 ms 2024-09-21T01:16:28.859 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c746d144', total seconds elapsed: 22.3 2024-09-21T01:16:28.859 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.8 2024-09-21T01:16:28.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:16:29.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 522692 3.88 MB/sec execute 536 sec latency 15.009 ms 2024-09-21T01:16:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:29.777 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:29 smithi039 ceph-mon[30415]: pgmap v792: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 2.3 MiB/s wr, 89 op/s 2024-09-21T01:16:29.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:29 smithi150 ceph-mon[34945]: pgmap v792: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 2.3 MiB/s wr, 89 op/s 2024-09-21T01:16:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:29 smithi160 ceph-mon[34696]: pgmap v792: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 2.3 MiB/s wr, 89 op/s 2024-09-21T01:16:29.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 -- 172.21.15.39:0/2900859393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85441039b0 msgr2=0x7f8544103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 --2- 172.21.15.39:0/2900859393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85441039b0 0x7f8544103dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8534009ca0 tx=0x7f853402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 -- 172.21.15.39:0/2900859393 shutdown_connections 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 --2- 172.21.15.39:0/2900859393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8544109c40 0x7f8544110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 --2- 172.21.15.39:0/2900859393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85441092f0 0x7f8544109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 --2- 172.21.15.39:0/2900859393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85441039b0 0x7f8544103dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 -- 172.21.15.39:0/2900859393 >> 172.21.15.39:0/2900859393 conn(0x7f85440ff440 msgr2=0x7f8544101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 -- 172.21.15.39:0/2900859393 shutdown_connections 2024-09-21T01:16:29.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.986+0000 7f8548c00640 1 -- 172.21.15.39:0/2900859393 wait complete. 2024-09-21T01:16:29.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 Processor -- start 2024-09-21T01:16:29.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 -- start start 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85441039b0 0x7f854419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85441092f0 0x7f854419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 0x7f854419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113950 con 0x7f85441092f0 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85441137d0 con 0x7f8544109c40 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.988+0000 7f8548c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113ad0 con 0x7f85441039b0 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 0x7f854419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 0x7f854419fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:57712/0 (socket says 172.21.15.39:57712) 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 -- 172.21.15.39:0/50331762 learned_addr learned my addr 172.21.15.39:0/50331762 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:29.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f853aa00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85441092f0 0x7f854419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:29.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543400640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85441039b0 0x7f854419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:29.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 -- 172.21.15.39:0/50331762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85441039b0 msgr2=0x7f854419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:29.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85441039b0 0x7f854419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:29.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 -- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85441092f0 msgr2=0x7f854419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85441092f0 0x7f854419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.989+0000 7f8543e00640 1 -- 172.21.15.39:0/50331762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85441a0290 con 0x7f8544109c40 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f853aa00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85441092f0 0x7f854419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f8543e00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 0x7f854419fb60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f853c00e9b0 tx=0x7f853c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c00cd80 con 0x7f8544109c40 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853c0035f0 con 0x7f8544109c40 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c010690 con 0x7f8544109c40 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85441a0580 con 0x7f8544109c40 2024-09-21T01:16:29.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.990+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85441a6ce0 con 0x7f8544109c40 2024-09-21T01:16:29.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.992+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8544103dc0 con 0x7f8544109c40 2024-09-21T01:16:29.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.994+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f853c004020 con 0x7f8544109c40 2024-09-21T01:16:29.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.996+0000 7f8540c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85280b6700 0x7f85280b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:29.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.996+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f853c0e7d70 con 0x7f8544109c40 2024-09-21T01:16:29.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.996+0000 7f8543400640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85280b6700 0x7f85280b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:29.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:29.997+0000 7f8543400640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85280b6700 0x7f85280b8c10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8534006010 tx=0x7f8534009820 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:30.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.000+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f853c0ade80 con 0x7f8544109c40 2024-09-21T01:16:30.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 523906 3.88 MB/sec execute 537 sec latency 15.766 ms 2024-09-21T01:16:30.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.308+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f854410ba10 con 0x7f8544109c40 2024-09-21T01:16:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.309+0000 7f8540c00640 1 -- 172.21.15.39:0/50331762 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11918 (secure 0 0 0) 0x7f853c0b2260 con 0x7f8544109c40 2024-09-21T01:16:30.311 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:16:30.311 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"btime":"2024-09-21T01:15:30:474236+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:15:29.524010+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:16:30.311 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 40 2024-09-21T01:16:30.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85280b6700 msgr2=0x7f85280b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:30.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85280b6700 0x7f85280b8c10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8534006010 tx=0x7f8534009820 comp rx=0 tx=0).stop 2024-09-21T01:16:30.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 msgr2=0x7f854419fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:30.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 0x7f854419fb60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f853c00e9b0 tx=0x7f853c00ee70 comp rx=0 tx=0).stop 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 shutdown_connections 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f85280b6700 0x7f85280b8c10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544109c40 0x7f854419fb60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85441092f0 0x7f854419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 --2- 172.21.15.39:0/50331762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f85441039b0 0x7f854419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 >> 172.21.15.39:0/50331762 conn(0x7f85440ff440 msgr2=0x7f8544101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 shutdown_connections 2024-09-21T01:16:30.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:30.312+0000 7f8548c00640 1 -- 172.21.15.39:0/50331762 wait complete. 2024-09-21T01:16:30.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:30 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/50331762' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:30.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:30 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/50331762' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:30.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:30 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/50331762' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:31.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 525973 3.90 MB/sec execute 538 sec latency 15.738 ms 2024-09-21T01:16:31.382 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0' 2024-09-21T01:16:31.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0 2024-09-21T01:16:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:31.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:31 smithi150 ceph-mon[34945]: pgmap v793: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 2.2 MiB/s wr, 87 op/s 2024-09-21T01:16:31.860 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:31 smithi160 ceph-mon[34696]: pgmap v793: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 2.2 MiB/s wr, 87 op/s 2024-09-21T01:16:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:31 smithi039 ceph-mon[30415]: pgmap v793: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 2.2 MiB/s wr, 87 op/s 2024-09-21T01:16:32.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 526678 3.90 MB/sec execute 539 sec latency 23.887 ms 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 -- 172.21.15.39:0/2329638184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0941049d0 msgr2=0x7ff094104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 --2- 172.21.15.39:0/2329638184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0941049d0 0x7ff094104de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff084009900 tx=0x7ff08402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 -- 172.21.15.39:0/2329638184 shutdown_connections 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 --2- 172.21.15.39:0/2329638184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 0x7ff094110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 --2- 172.21.15.39:0/2329638184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff094105320 0x7ff09410d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 --2- 172.21.15.39:0/2329638184 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0941049d0 0x7ff094104de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 -- 172.21.15.39:0/2329638184 >> 172.21.15.39:0/2329638184 conn(0x7ff094079760 msgr2=0x7ff094079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 -- 172.21.15.39:0/2329638184 shutdown_connections 2024-09-21T01:16:32.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.454+0000 7ff09a800640 1 -- 172.21.15.39:0/2329638184 wait complete. 2024-09-21T01:16:32.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.456+0000 7ff09a800640 1 Processor -- start 2024-09-21T01:16:32.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.456+0000 7ff09a800640 1 -- start start 2024-09-21T01:16:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff09a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0941049d0 0x7ff09419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff09a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 0x7ff09419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff09a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 0x7ff0941a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff09a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff094113a90 con 0x7ff0941049d0 2024-09-21T01:16:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff09a800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff094113910 con 0x7ff09410de30 2024-09-21T01:16:32.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff09a800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff094113c10 con 0x7ff094105320 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 0x7ff09419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 0x7ff09419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36482/0 (socket says 172.21.15.39:36482) 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 -- 172.21.15.39:0/2379329112 learned_addr learned my addr 172.21.15.39:0/2379329112 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff093400640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0941049d0 0x7ff09419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff093e00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 0x7ff0941a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 -- 172.21.15.39:0/2379329112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 msgr2=0x7ff0941a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 0x7ff0941a1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 -- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0941049d0 msgr2=0x7ff09419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0941049d0 0x7ff09419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff08be00640 1 -- 172.21.15.39:0/2379329112 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0941a2610 con 0x7ff094105320 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.457+0000 7ff093e00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 0x7ff0941a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:32.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.458+0000 7ff093400640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0941049d0 0x7ff09419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:32.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.458+0000 7ff08be00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 0x7ff09419cc40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00e9f0 tx=0x7ff07c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:32.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.458+0000 7ff090c00640 1 -- 172.21.15.39:0/2379329112 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c00ccc0 con 0x7ff094105320 2024-09-21T01:16:32.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.458+0000 7ff090c00640 1 -- 172.21.15.39:0/2379329112 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff07c0108e0 con 0x7ff094105320 2024-09-21T01:16:32.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.458+0000 7ff090c00640 1 -- 172.21.15.39:0/2379329112 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c0187c0 con 0x7ff094105320 2024-09-21T01:16:32.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.458+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0941a2900 con 0x7ff094105320 2024-09-21T01:16:32.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.459+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0941a9060 con 0x7ff094105320 2024-09-21T01:16:32.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.463+0000 7ff090c00640 1 -- 172.21.15.39:0/2379329112 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff07c010430 con 0x7ff094105320 2024-09-21T01:16:32.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.463+0000 7ff0a1486740 1 Processor -- start 2024-09-21T01:16:32.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.463+0000 7ff0a1486740 1 -- start start 2024-09-21T01:16:32.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774d19840 0x5597750c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf0980 0x5597750ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf8b60 0x5597750ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff0a1486740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559775036ca0 con 0x559774d19840 2024-09-21T01:16:32.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff0a1486740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5597750260d0 con 0x559774cf8b60 2024-09-21T01:16:32.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff0a1486740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559775029ee0 con 0x559774cf0980 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff093e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774d19840 0x5597750c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff093e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774d19840 0x5597750c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49074/0 (socket says 172.21.15.39:49074) 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff093e00640 1 -- 172.21.15.39:0/3120346487 learned_addr learned my addr 172.21.15.39:0/3120346487 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff08be00640 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf8b60 0x5597750ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.464+0000 7ff093400640 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf0980 0x5597750ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff093400640 1 -- 172.21.15.39:0/3120346487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf8b60 msgr2=0x5597750ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff093400640 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf8b60 0x5597750ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff093400640 1 -- 172.21.15.39:0/3120346487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774d19840 msgr2=0x5597750c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff093400640 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774d19840 0x5597750c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff093400640 1 -- 172.21.15.39:0/3120346487 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559774fd8400 con 0x559774cf0980 2024-09-21T01:16:32.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff090c00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0780b6780 0x7ff0780b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff093400640 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf0980 0x5597750ca0a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff084006010 tx=0x7ff084032c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f912c66c1afa70d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:32.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.465+0000 7ff090c00640 1 -- 172.21.15.39:0/2379329112 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7ff07c0e7f20 con 0x7ff094105320 2024-09-21T01:16:32.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.466+0000 7ff088c00640 1 -- 172.21.15.39:0/3120346487 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08403f070 con 0x559774cf0980 2024-09-21T01:16:32.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.466+0000 7ff088c00640 1 -- 172.21.15.39:0/3120346487 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0840316a0 con 0x559774cf0980 2024-09-21T01:16:32.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.466+0000 7ff088c00640 1 -- 172.21.15.39:0/3120346487 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084035070 con 0x559774cf0980 2024-09-21T01:16:32.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.466+0000 7ff093e00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0780b6780 0x7ff0780b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 -- 172.21.15.39:0/3120346487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf0980 msgr2=0x5597750ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf0980 0x5597750ca0a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff084006010 tx=0x7ff084032c50 comp rx=0 tx=0).stop 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff093e00640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0780b6780 0x7ff0780b8c90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff08c005c40 tx=0x7ff08c009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 -- 172.21.15.39:0/3120346487 shutdown_connections 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf8b60 0x5597750ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf0980 0x5597750ca0a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3120346487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774d19840 0x5597750c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 -- 172.21.15.39:0/3120346487 >> 172.21.15.39:0/3120346487 conn(0x559774d0d990 msgr2=0x5597750c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 -- 172.21.15.39:0/3120346487 shutdown_connections 2024-09-21T01:16:32.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.467+0000 7ff0a1486740 1 -- 172.21.15.39:0/3120346487 wait complete. 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.468+0000 7ff0a1486740 1 -- start start 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.468+0000 7ff0a1486740 1 Processor -- start 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750d52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750d57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559774e877a0 con 0x559774cf0980 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559774fcba10 con 0x559774d19840 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5597750d82e0 con 0x559774cf8b60 2024-09-21T01:16:32.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff0a1486740 10 client.0 _ll_register_callbacks cb 0x559774fdbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:32.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff093e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff093e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:57756/0 (socket says 172.21.15.39:57756) 2024-09-21T01:16:32.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff093e00640 1 -- 172.21.15.39:0/865348067 learned_addr learned my addr 172.21.15.39:0/865348067 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:32.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.469+0000 7ff093400640 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750d52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff093e00640 1 -- 172.21.15.39:0/865348067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 msgr2=0x5597750d57e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff093e00640 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750d57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff093e00640 1 -- 172.21.15.39:0/865348067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 msgr2=0x5597750d52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff093e00640 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750d52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff093e00640 1 -- 172.21.15.39:0/865348067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559774d93b90 con 0x559774d19840 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff093e00640 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e15d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff08c010ba0 tx=0x7ff08c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f3f0125e5ce8157 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.470+0000 7ff081600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:16:32.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff081600640 1 -- 172.21.15.39:0/865348067 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c004d20 con 0x559774d19840 2024-09-21T01:16:32.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff081600640 1 -- 172.21.15.39:0/865348067 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7ff08c01e9a0 con 0x559774d19840 2024-09-21T01:16:32.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0840316a0 con 0x559774d19840 2024-09-21T01:16:32.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff081600640 1 -- 172.21.15.39:0/865348067 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff08c01eba0 con 0x559774d19840 2024-09-21T01:16:32.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff0a1486740 10 client.26081 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:32.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5597750e40d0 con 0x559774d19840 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.471+0000 7ff081600640 1 -- 172.21.15.39:0/865348067 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0840316a0 con 0x559774d19840 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff081600640 1 -- 172.21.15.39:0/865348067 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7ff08c01c410 con 0x559774d19840 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff0a1486740 10 client.26081 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff0a1486740 10 client.26081 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff0a1486740 20 client.26081 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff0a1486740 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cd81c0 0x5597750e7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff0a1486740 4 client.26081 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.472+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559775038bf0 con 0x559774cd81c0 2024-09-21T01:16:32.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.473+0000 7ff093400640 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cd81c0 0x5597750e7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.473+0000 7ff093400640 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cd81c0 0x5597750e7730 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ada21ace24aa6cc server_cookie=3e2e9e99d4a36999 in_seq=0 out_seq=0 2024-09-21T01:16:32.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.473+0000 7ff081600640 10 client.26081 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:32.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.474+0000 7ff081600640 1 -- 172.21.15.39:0/865348067 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559775038bf0 con 0x559774cd81c0 2024-09-21T01:16:32.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.474+0000 7ff081600640 10 client.26081 handle_command_reply: tid=0 2024-09-21T01:16:32.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.474+0000 7ff0a1486740 1 client.26081 shutdown 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 msgr2=0x5597750e15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e15d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff08c010ba0 tx=0x7ff08c004a40 comp rx=0 tx=0).stop 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 shutdown_connections 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cd81c0 0x5597750e7730 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e15d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750d57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 --2- 172.21.15.39:0/865348067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750d52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 >> 172.21.15.39:0/865348067 conn(0x559774d0d990 msgr2=0x5597750bd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:32.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.475+0000 7ff081600640 0 client.26081 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:32.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.476+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 shutdown_connections 2024-09-21T01:16:32.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.476+0000 7ff0a1486740 1 -- 172.21.15.39:0/865348067 wait complete. 2024-09-21T01:16:32.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.476+0000 7ff0a1486740 20 client.26081 trim_cache size 0 max 16384 2024-09-21T01:16:32.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.498+0000 7ff0a1486740 1 Processor -- start 2024-09-21T01:16:32.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.498+0000 7ff0a1486740 1 -- start start 2024-09-21T01:16:32.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff0a1486740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0941b17a0 con 0x559774cf0980 2024-09-21T01:16:32.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff0a1486740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0941aefe0 con 0x559774d19840 2024-09-21T01:16:32.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff0a1486740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0941ace90 con 0x559774cf8b60 2024-09-21T01:16:32.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff093400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff093e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff093e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:57764/0 (socket says 172.21.15.39:57764) 2024-09-21T01:16:32.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff093e00640 1 -- 172.21.15.39:0/425969685 learned_addr learned my addr 172.21.15.39:0/425969685 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:32.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.499+0000 7ff08be00640 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff093e00640 1 -- 172.21.15.39:0/425969685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 msgr2=0x5597750e14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff093e00640 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750e14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff093e00640 1 -- 172.21.15.39:0/425969685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 msgr2=0x5597750cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff093e00640 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff093e00640 1 -- 172.21.15.39:0/425969685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597750c8810 con 0x559774d19840 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff093e00640 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e1a10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff08c012f40 tx=0x7ff08c009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=474d76dbc037fdaf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.500+0000 7ff083e00640 1 -- 172.21.15.39:0/425969685 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c01ccc0 con 0x559774d19840 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.501+0000 7ff083e00640 1 -- 172.21.15.39:0/425969685 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08c01e7f0 con 0x559774d19840 2024-09-21T01:16:32.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.501+0000 7ff083e00640 1 -- 172.21.15.39:0/425969685 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c036c70 con 0x559774d19840 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 -- 172.21.15.39:0/425969685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 msgr2=0x5597750e1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e1a10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff08c012f40 tx=0x7ff08c009ad0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 -- 172.21.15.39:0/425969685 shutdown_connections 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774d19840 0x5597750e1a10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774cf8b60 0x5597750e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 --2- 172.21.15.39:0/425969685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf0980 0x5597750cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 -- 172.21.15.39:0/425969685 >> 172.21.15.39:0/425969685 conn(0x559774d0d990 msgr2=0x5597750cab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 -- 172.21.15.39:0/425969685 shutdown_connections 2024-09-21T01:16:32.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.502+0000 7ff0a1486740 1 -- 172.21.15.39:0/425969685 wait complete. 2024-09-21T01:16:32.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 -- start start 2024-09-21T01:16:32.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 Processor -- start 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf0980 0x5597750d9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf8b60 0x5597750d9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 0x5597750d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0941ad4e0 con 0x559774cf8b60 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0941ae910 con 0x559774cf0980 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.503+0000 7ff0a1486740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0941adb70 con 0x559774d19840 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff0a1486740 10 client.0 _ll_register_callbacks cb 0x7ff08c009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 0x5597750d9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 0x5597750d9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36508/0 (socket says 172.21.15.39:36508) 2024-09-21T01:16:32.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 -- 172.21.15.39:0/3320245690 learned_addr learned my addr 172.21.15.39:0/3320245690 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093400640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf0980 0x5597750d9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 -- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf0980 msgr2=0x5597750d9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf0980 0x5597750d9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 -- 172.21.15.39:0/3320245690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf8b60 msgr2=0x5597750d9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf8b60 0x5597750d9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.504+0000 7ff093e00640 1 -- 172.21.15.39:0/3320245690 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559775151130 con 0x559774d19840 2024-09-21T01:16:32.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.505+0000 7ff093400640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf0980 0x5597750d9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.505+0000 7ff093e00640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 0x5597750d9fd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009690 tx=0x7ff08c010b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e8b3f89d5c5c5f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.505+0000 7ff081600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.505+0000 7ff081600640 1 -- 172.21.15.39:0/3320245690 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c004c10 con 0x559774d19840 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.505+0000 7ff081600640 1 -- 172.21.15.39:0/3320245690 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7ff08c004db0 con 0x559774d19840 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.506+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff08c01e7f0 con 0x559774d19840 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.506+0000 7ff081600640 1 -- 172.21.15.39:0/3320245690 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff08c005030 con 0x559774d19840 2024-09-21T01:16:32.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.506+0000 7ff0a1486740 10 client.26077 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.506+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x559775152390 con 0x559774d19840 2024-09-21T01:16:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff081600640 1 -- 172.21.15.39:0/3320245690 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08c01e7f0 con 0x559774d19840 2024-09-21T01:16:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff081600640 1 -- 172.21.15.39:0/3320245690 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7ff08c049030 con 0x559774d19840 2024-09-21T01:16:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff0a1486740 10 client.26077 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff0a1486740 10 client.26077 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff0a1486740 20 client.26077 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:32.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cad570 0x559775154b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:32.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff0a1486740 4 client.26077 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}] 2024-09-21T01:16:32.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.507+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}) -- 0x5597750346a0 con 0x559774cad570 2024-09-21T01:16:32.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.508+0000 7ff093400640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cad570 0x559775154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:32.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.508+0000 7ff081600640 10 client.26077 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:32.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.508+0000 7ff093400640 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cad570 0x559775154b80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd17eb91e679a3f3 server_cookie=f7ddc6eea2eb9e79 in_seq=0 out_seq=0 2024-09-21T01:16:32.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff081600640 1 -- 172.21.15.39:0/3320245690 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5597750346a0 con 0x559774cad570 2024-09-21T01:16:32.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff081600640 10 client.26077 handle_command_reply: tid=0 2024-09-21T01:16:32.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 client.26077 shutdown 2024-09-21T01:16:32.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 msgr2=0x5597750d9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 0x5597750d9fd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009690 tx=0x7ff08c010b60 comp rx=0 tx=0).stop 2024-09-21T01:16:32.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 shutdown_connections 2024-09-21T01:16:32.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559774cad570 0x559775154b80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559774d19840 0x5597750d9fd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559774cf8b60 0x5597750d9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 --2- 172.21.15.39:0/3320245690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559774cf0980 0x5597750d9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 >> 172.21.15.39:0/3320245690 conn(0x559774d0d990 msgr2=0x5597750cab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.731+0000 7ff081600640 0 client.26077 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.732+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 shutdown_connections 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.732+0000 7ff0a1486740 1 -- 172.21.15.39:0/3320245690 wait complete. 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.732+0000 7ff0a1486740 20 client.26077 trim_cache size 0 max 16384 2024-09-21T01:16:32.734 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 40, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 50, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "f65cb898": { 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 50, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 137.8, 2024-09-21T01:16:32.735 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:32.736 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0780b6780 msgr2=0x7ff0780b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0780b6780 0x7ff0780b8c90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff08c005c40 tx=0x7ff08c009500 comp rx=0 tx=0).stop 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 msgr2=0x7ff09419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 0x7ff09419cc40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00e9f0 tx=0x7ff07c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 shutdown_connections 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff0780b6780 0x7ff0780b8c90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff09410de30 0x7ff0941a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff094105320 0x7ff09419cc40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 --2- 172.21.15.39:0/2379329112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0941049d0 0x7ff09419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 >> 172.21.15.39:0/2379329112 conn(0x7ff094079760 msgr2=0x7ff094109c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 shutdown_connections 2024-09-21T01:16:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:32.733+0000 7ff09a800640 1 -- 172.21.15.39:0/2379329112 wait complete. 2024-09-21T01:16:33.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.90 MB/sec execute 540 sec latency 454.606 ms 2024-09-21T01:16:33.648 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f65cb898', quiesce took 4.8 seconds. Will release after: 13.0 2024-09-21T01:16:33.805 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:33 smithi039 ceph-mon[30415]: pgmap v794: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 2.0 MiB/s wr, 79 op/s 2024-09-21T01:16:33.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:33 smithi150 ceph-mon[34945]: pgmap v794: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 2.0 MiB/s wr, 79 op/s 2024-09-21T01:16:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:33 smithi160 ceph-mon[34696]: pgmap v794: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 2.0 MiB/s wr, 79 op/s 2024-09-21T01:16:34.199 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.89 MB/sec execute 541 sec latency 1454.739 ms 2024-09-21T01:16:35.200 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.88 MB/sec execute 542 sec latency 2454.892 ms 2024-09-21T01:16:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:35 smithi150 ceph-mon[34945]: pgmap v795: 129 pgs: 129 active+clean; 433 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T01:16:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:35 smithi160 ceph-mon[34696]: pgmap v795: 129 pgs: 129 active+clean; 433 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T01:16:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:35 smithi039 ceph-mon[30415]: pgmap v795: 129 pgs: 129 active+clean; 433 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T01:16:36.200 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.88 MB/sec execute 543 sec latency 3455.051 ms 2024-09-21T01:16:37.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.87 MB/sec execute 544 sec latency 4455.183 ms 2024-09-21T01:16:37.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:37 smithi150 ceph-mon[34945]: pgmap v796: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 906 KiB/s wr, 92 op/s 2024-09-21T01:16:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:37 smithi160 ceph-mon[34696]: pgmap v796: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 906 KiB/s wr, 92 op/s 2024-09-21T01:16:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:37 smithi039 ceph-mon[30415]: pgmap v796: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 906 KiB/s wr, 92 op/s 2024-09-21T01:16:38.200 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.86 MB/sec execute 545 sec latency 5455.274 ms 2024-09-21T01:16:39.200 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.85 MB/sec execute 546 sec latency 6455.345 ms 2024-09-21T01:16:39.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:39 smithi150 ceph-mon[34945]: pgmap v797: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 944 KiB/s wr, 95 op/s 2024-09-21T01:16:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:39 smithi160 ceph-mon[34696]: pgmap v797: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 944 KiB/s wr, 95 op/s 2024-09-21T01:16:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:39 smithi039 ceph-mon[30415]: pgmap v797: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 944 KiB/s wr, 95 op/s 2024-09-21T01:16:40.200 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.85 MB/sec execute 547 sec latency 7455.443 ms 2024-09-21T01:16:41.200 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.84 MB/sec execute 548 sec latency 8455.548 ms 2024-09-21T01:16:41.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:41 smithi160 ceph-mon[34696]: pgmap v798: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 842 KiB/s wr, 80 op/s 2024-09-21T01:16:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:41 smithi039 ceph-mon[30415]: pgmap v798: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 842 KiB/s wr, 80 op/s 2024-09-21T01:16:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:41 smithi150 ceph-mon[34945]: pgmap v798: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 842 KiB/s wr, 80 op/s 2024-09-21T01:16:42.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.83 MB/sec execute 549 sec latency 9455.689 ms 2024-09-21T01:16:43.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.83 MB/sec execute 550 sec latency 10455.792 ms 2024-09-21T01:16:43.842 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:43 smithi039 ceph-mon[30415]: pgmap v799: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 831 KiB/s wr, 77 op/s 2024-09-21T01:16:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:43 smithi160 ceph-mon[34696]: pgmap v799: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 831 KiB/s wr, 77 op/s 2024-09-21T01:16:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:43 smithi150 ceph-mon[34945]: pgmap v799: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 831 KiB/s wr, 77 op/s 2024-09-21T01:16:44.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.82 MB/sec execute 551 sec latency 11455.866 ms 2024-09-21T01:16:45.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.81 MB/sec execute 552 sec latency 12455.995 ms 2024-09-21T01:16:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:45 smithi160 ceph-mon[34696]: pgmap v800: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 816 KiB/s wr, 77 op/s 2024-09-21T01:16:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:45 smithi039 ceph-mon[30415]: pgmap v800: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 816 KiB/s wr, 77 op/s 2024-09-21T01:16:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:45 smithi150 ceph-mon[34945]: pgmap v800: 129 pgs: 129 active+clean; 435 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 816 KiB/s wr, 77 op/s 2024-09-21T01:16:46.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.81 MB/sec execute 553 sec latency 13456.099 ms 2024-09-21T01:16:46.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:16:47.043 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:47.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.80 MB/sec execute 554 sec latency 14456.239 ms 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.823+0000 7f222d800640 1 -- 172.21.15.39:0/403496123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22281039b0 msgr2=0x7f2228103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.823+0000 7f222d800640 1 --2- 172.21.15.39:0/403496123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22281039b0 0x7f2228103dc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2218009900 tx=0x7f221802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 -- 172.21.15.39:0/403496123 shutdown_connections 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 --2- 172.21.15.39:0/403496123 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2228109c40 0x7f2228110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 --2- 172.21.15.39:0/403496123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281092f0 0x7f2228109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 --2- 172.21.15.39:0/403496123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22281039b0 0x7f2228103dc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 -- 172.21.15.39:0/403496123 >> 172.21.15.39:0/403496123 conn(0x7f22280ff440 msgr2=0x7f2228101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 -- 172.21.15.39:0/403496123 shutdown_connections 2024-09-21T01:16:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.824+0000 7f222d800640 1 -- 172.21.15.39:0/403496123 wait complete. 2024-09-21T01:16:47.828 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:47 smithi039 ceph-mon[30415]: pgmap v801: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 343 KiB/s wr, 28 op/s 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.825+0000 7f222d800640 1 Processor -- start 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.825+0000 7f222d800640 1 -- start start 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222d800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281039b0 0x7f222819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222d800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 0x7f222819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2228109c40 0x7f22281a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2228113950 con 0x7f2228109c40 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222d800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f22281137d0 con 0x7f22281039b0 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222d800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2228113ad0 con 0x7f22281092f0 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f2227400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 0x7f222819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f222ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2228109c40 0x7f22281a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f2227400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 0x7f222819cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43088/0 (socket says 172.21.15.39:43088) 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f2227e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281039b0 0x7f222819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.826+0000 7f2227400640 1 -- 172.21.15.39:0/3103945486 learned_addr learned my addr 172.21.15.39:0/3103945486 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227400640 1 -- 172.21.15.39:0/3103945486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281039b0 msgr2=0x7f222819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227400640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281039b0 0x7f222819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227400640 1 -- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2228109c40 msgr2=0x7f22281a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227400640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2228109c40 0x7f22281a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227400640 1 -- 172.21.15.39:0/3103945486 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22281a24b0 con 0x7f22281092f0 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227e00640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281039b0 0x7f222819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:47.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f222ce00640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2228109c40 0x7f22281a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:47.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.827+0000 7f2227400640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 0x7f222819cae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f221400e970 tx=0x7f221400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:47.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.828+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221400cd80 con 0x7f22281092f0 2024-09-21T01:16:47.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.828+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22140035f0 con 0x7f22281092f0 2024-09-21T01:16:47.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.828+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22281a27a0 con 0x7f22281092f0 2024-09-21T01:16:47.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.828+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214010690 con 0x7f22281092f0 2024-09-21T01:16:47.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.828+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22281a8f00 con 0x7f22281092f0 2024-09-21T01:16:47.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.829+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2228103dc0 con 0x7f22281092f0 2024-09-21T01:16:47.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.832+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2214003aa0 con 0x7f22281092f0 2024-09-21T01:16:47.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.833+0000 7f2224c00640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f220c0b6680 0x7f220c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:47.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.833+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f22140e7e70 con 0x7f22281092f0 2024-09-21T01:16:47.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.833+0000 7f2227e00640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f220c0b6680 0x7f220c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:47.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.834+0000 7f2227e00640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f220c0b6680 0x7f220c0b8b90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2218005ab0 tx=0x7f2218005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:47.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:47.835+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f22140adf80 con 0x7f22281092f0 2024-09-21T01:16:47.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:47 smithi160 ceph-mon[34696]: pgmap v801: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 343 KiB/s wr, 28 op/s 2024-09-21T01:16:47.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:47 smithi150 ceph-mon[34945]: pgmap v801: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 343 KiB/s wr, 28 op/s 2024-09-21T01:16:48.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.069+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f222819d670 con 0x7f22281092f0 2024-09-21T01:16:48.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.070+0000 7f2224c00640 1 -- 172.21.15.39:0/3103945486 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11918 (secure 0 0 0) 0x7f222819d670 con 0x7f22281092f0 2024-09-21T01:16:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:16:48.077 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"btime":"2024-09-21T01:15:30:474236+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:15:29.524010+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:16:48.077 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 40 2024-09-21T01:16:48.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f220c0b6680 msgr2=0x7f220c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:48.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f220c0b6680 0x7f220c0b8b90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2218005ab0 tx=0x7f2218005a20 comp rx=0 tx=0).stop 2024-09-21T01:16:48.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 msgr2=0x7f222819cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:48.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 0x7f222819cae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f221400e970 tx=0x7f221400ee30 comp rx=0 tx=0).stop 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 shutdown_connections 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2228109c40 0x7f22281a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22281092f0 0x7f222819cae0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f220c0b6680 0x7f220c0b8b90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 --2- 172.21.15.39:0/3103945486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22281039b0 0x7f222819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 >> 172.21.15.39:0/3103945486 conn(0x7f22280ff440 msgr2=0x7f2228108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 shutdown_connections 2024-09-21T01:16:48.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:48.073+0000 7f222d800640 1 -- 172.21.15.39:0/3103945486 wait complete. 2024-09-21T01:16:48.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.79 MB/sec execute 555 sec latency 15456.386 ms 2024-09-21T01:16:48.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:48 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3103945486' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:48.899 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:48 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3103945486' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:48.938 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id f65cb898 --release --await' 2024-09-21T01:16:48.938 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id f65cb898 --release --await 2024-09-21T01:16:49.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:48 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3103945486' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:16:49.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527167 3.79 MB/sec execute 556 sec latency 16456.445 ms 2024-09-21T01:16:49.319 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:16:49.711 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:49 smithi039 ceph-mon[30415]: pgmap v802: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 615 KiB/s wr, 27 op/s 2024-09-21T01:16:49.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:49 smithi160 ceph-mon[34696]: pgmap v802: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 615 KiB/s wr, 27 op/s 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.996+0000 7f067de00640 1 -- 172.21.15.39:0/82874516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 msgr2=0x7f067810bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.996+0000 7f067de00640 1 --2- 172.21.15.39:0/82874516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 0x7f067810bad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0664009900 tx=0x7f066402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 -- 172.21.15.39:0/82874516 shutdown_connections 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 --2- 172.21.15.39:0/82874516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f0678112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 --2- 172.21.15.39:0/82874516 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 0x7f067810bad0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 --2- 172.21.15.39:0/82874516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0678105d80 0x7f0678106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:49.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 -- 172.21.15.39:0/82874516 >> 172.21.15.39:0/82874516 conn(0x7f0678101810 msgr2=0x7f0678103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:49.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 -- 172.21.15.39:0/82874516 shutdown_connections 2024-09-21T01:16:49.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.997+0000 7f067de00640 1 -- 172.21.15.39:0/82874516 wait complete. 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 Processor -- start 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 -- start start 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0678105d80 0x7f067819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 0x7f067819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f06781a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0678115d20 con 0x7f067810c010 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0678115ba0 con 0x7f0678105d80 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.998+0000 7f067de00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0678115ea0 con 0x7f067810b6c0 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f06781a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0676000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 0x7f067819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0676a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0678105d80 0x7f067819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f06781a1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56266/0 (socket says 172.21.15.39:56266) 2024-09-21T01:16:50.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 -- 172.21.15.39:0/3453866917 learned_addr learned my addr 172.21.15.39:0/3453866917 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:50.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 -- 172.21.15.39:0/3453866917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 msgr2=0x7f067819cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 0x7f067819cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 -- 172.21.15.39:0/3453866917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0678105d80 msgr2=0x7f067819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0678105d80 0x7f067819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 -- 172.21.15.39:0/3453866917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06781a2640 con 0x7f067810c010 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:49.999+0000 7f0677400640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f06781a1f10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f067000efd0 tx=0x7f067000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.000+0000 7f067ca00640 1 -- 172.21.15.39:0/3453866917 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670003b70 con 0x7f067810c010 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.000+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06781a2930 con 0x7f067810c010 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.000+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06781a9090 con 0x7f067810c010 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.000+0000 7f067ca00640 1 -- 172.21.15.39:0/3453866917 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0670003d10 con 0x7f067810c010 2024-09-21T01:16:50.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.000+0000 7f067ca00640 1 -- 172.21.15.39:0/3453866917 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06700108c0 con 0x7f067810c010 2024-09-21T01:16:50.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.003+0000 7f0684be3740 1 Processor -- start 2024-09-21T01:16:50.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.003+0000 7f067ca00640 1 -- 172.21.15.39:0/3453866917 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f0670010a60 con 0x7f067810c010 2024-09-21T01:16:50.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 -- start start 2024-09-21T01:16:50.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe9a520 0x55b7702411c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770241700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770241c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b7701b8290 con 0x55b76fe79840 2024-09-21T01:16:50.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55b7701a76c0 con 0x55b76fe71660 2024-09-21T01:16:50.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.004+0000 7f0684be3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b7701ab4d0 con 0x55b76fe9a520 2024-09-21T01:16:50.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe9a520 0x55b7702411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe9a520 0x55b7702411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43128/0 (socket says 172.21.15.39:43128) 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676000640 1 -- 172.21.15.39:0/2871545655 learned_addr learned my addr 172.21.15.39:0/2871545655 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676a00640 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770241700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676000640 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770241c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676a00640 1 -- 172.21.15.39:0/2871545655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe9a520 msgr2=0x55b7702411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676a00640 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe9a520 0x55b7702411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676a00640 1 -- 172.21.15.39:0/2871545655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 msgr2=0x55b770241c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676a00640 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770241c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676a00640 1 -- 172.21.15.39:0/2871545655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b770134460 con 0x55b76fe71660 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f0676000640 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770241c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.005+0000 7f067ca00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f065c0b6700 0x7f065c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.006+0000 7f0676a00640 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770241700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0668009c00 tx=0x7f0668011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddda70c0e8daced8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:50.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.006+0000 7f067ca00640 1 -- 172.21.15.39:0/3453866917 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f06700e80c0 con 0x7f067810c010 2024-09-21T01:16:50.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.006+0000 7f0676000640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f065c0b6700 0x7f065c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.006+0000 7f0663e00640 1 -- 172.21.15.39:0/2871545655 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668004230 con 0x55b76fe71660 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.006+0000 7f0663e00640 1 -- 172.21.15.39:0/2871545655 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06680043d0 con 0x55b76fe71660 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.006+0000 7f0663e00640 1 -- 172.21.15.39:0/2871545655 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668004c70 con 0x55b76fe71660 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 -- 172.21.15.39:0/2871545655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 msgr2=0x55b770241700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770241700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0668009c00 tx=0x7f0668011810 comp rx=0 tx=0).stop 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0676000640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f065c0b6700 0x7f065c0b8c10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0664031440 tx=0x7f0664005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 -- 172.21.15.39:0/2871545655 shutdown_connections 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770241c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770241700 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 --2- 172.21.15.39:0/2871545655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe9a520 0x55b7702411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 -- 172.21.15.39:0/2871545655 >> 172.21.15.39:0/2871545655 conn(0x55b76fe8e910 msgr2=0x55b77023f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 -- 172.21.15.39:0/2871545655 shutdown_connections 2024-09-21T01:16:50.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.007+0000 7f0684be3740 1 -- 172.21.15.39:0/2871545655 wait complete. 2024-09-21T01:16:50.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 -- start start 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 Processor -- start 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe71660 0x55b77024cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b77024d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe9a520 0x55b770258d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b770008bb0 con 0x55b76fe79840 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55b7701426c0 con 0x55b76fe9a520 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.008+0000 7f0684be3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55b77024f9a0 con 0x55b76fe71660 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0684be3740 10 client.0 _ll_register_callbacks cb 0x55b77015d490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b77024d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b77024d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56290/0 (socket says 172.21.15.39:56290) 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676a00640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe71660 0x55b77024cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0677400640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe9a520 0x55b770258d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676000640 1 -- 172.21.15.39:0/3387025730 learned_addr learned my addr 172.21.15.39:0/3387025730 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676a00640 1 -- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe9a520 msgr2=0x55b770258d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676a00640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe9a520 0x55b770258d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676a00640 1 -- 172.21.15.39:0/3387025730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 msgr2=0x55b77024d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676a00640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b77024d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.009+0000 7f0676a00640 1 -- 172.21.15.39:0/3387025730 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b76ff05b00 con 0x55b76fe71660 2024-09-21T01:16:50.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.010+0000 7f0677400640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe9a520 0x55b770258d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:50.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.010+0000 7f0676a00640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe71660 0x55b77024cda0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0668009bd0 tx=0x7f0668019b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cac5fb8bed11e9b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:50.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.010+0000 7f0660c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:16:50.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.010+0000 7f0660c00640 1 -- 172.21.15.39:0/3387025730 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668028020 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.010+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06680043d0 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.010+0000 7f0660c00640 1 -- 172.21.15.39:0/3387025730 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f0668024620 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0660c00640 1 -- 172.21.15.39:0/3387025730 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0668033050 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 10 client.26095 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b76ff7c960 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0660c00640 1 -- 172.21.15.39:0/3387025730 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06680043d0 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0660c00640 1 -- 172.21.15.39:0/3387025730 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f0668018070 con 0x55b76fe71660 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 10 client.26095 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 10 client.26095 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 20 client.26095 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe58ea0 0x55b77025eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 4 client.26095 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.011+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7701ba1e0 con 0x55b76fe58ea0 2024-09-21T01:16:50.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.012+0000 7f0676000640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe58ea0 0x55b77025eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.012+0000 7f0676000640 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe58ea0 0x55b77025eba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e116c98a39bb98e9 server_cookie=40bb6e3f9f99a5da in_seq=0 out_seq=0 2024-09-21T01:16:50.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.012+0000 7f0660c00640 10 client.26095 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:50.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.013+0000 7f0660c00640 1 -- 172.21.15.39:0/3387025730 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b7701ba1e0 con 0x55b76fe58ea0 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.013+0000 7f0660c00640 10 client.26095 handle_command_reply: tid=0 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.013+0000 7f0684be3740 1 client.26095 shutdown 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe71660 msgr2=0x55b77024cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe71660 0x55b77024cda0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0668009bd0 tx=0x7f0668019b70 comp rx=0 tx=0).stop 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 shutdown_connections 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe58ea0 0x55b77025eba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe9a520 0x55b770258d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b77024d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 --2- 172.21.15.39:0/3387025730 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe71660 0x55b77024cda0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0660c00640 0 client.26095 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 >> 172.21.15.39:0/3387025730 conn(0x55b76fe8e910 msgr2=0x55b770244350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 shutdown_connections 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 1 -- 172.21.15.39:0/3387025730 wait complete. 2024-09-21T01:16:50.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.014+0000 7f0684be3740 20 client.26095 trim_cache size 0 max 16384 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.032+0000 7f0684be3740 1 Processor -- start 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.032+0000 7f0684be3740 1 -- start start 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0684be3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe8e910 0x55b77023ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0684be3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770258c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0684be3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770259160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0684be3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f06580129a0 con 0x55b76fe79840 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0684be3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06580101e0 con 0x55b76fe71660 2024-09-21T01:16:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0684be3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f065800e050 con 0x55b76fe8e910 2024-09-21T01:16:50.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770259160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0676000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770258c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770259160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56302/0 (socket says 172.21.15.39:56302) 2024-09-21T01:16:50.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 -- 172.21.15.39:0/1000481893 learned_addr learned my addr 172.21.15.39:0/1000481893 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0676a00640 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe8e910 0x55b77023ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 -- 172.21.15.39:0/1000481893 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe8e910 msgr2=0x55b77023ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe8e910 0x55b77023ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 -- 172.21.15.39:0/1000481893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 msgr2=0x55b770258c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770258c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.033+0000 7f0677400640 1 -- 172.21.15.39:0/1000481893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7702483c0 con 0x55b76fe79840 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.034+0000 7f0676a00640 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe8e910 0x55b77023ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.034+0000 7f0676000640 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770258c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.034+0000 7f0677400640 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770259160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f06700ad830 tx=0x7f0670095040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57dbe61cec52bc1a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.034+0000 7f0663400640 1 -- 172.21.15.39:0/1000481893 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06700aee60 con 0x55b76fe79840 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.034+0000 7f0663400640 1 -- 172.21.15.39:0/1000481893 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06700ad640 con 0x55b76fe79840 2024-09-21T01:16:50.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.034+0000 7f0663400640 1 -- 172.21.15.39:0/1000481893 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06700acd90 con 0x55b76fe79840 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 -- 172.21.15.39:0/1000481893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 msgr2=0x55b770259160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770259160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f06700ad830 tx=0x7f0670095040 comp rx=0 tx=0).stop 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 -- 172.21.15.39:0/1000481893 shutdown_connections 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe79840 0x55b770259160 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe71660 0x55b770258c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 --2- 172.21.15.39:0/1000481893 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe8e910 0x55b77023ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 -- 172.21.15.39:0/1000481893 >> 172.21.15.39:0/1000481893 conn(0x55b77023f0f0 msgr2=0x55b77024cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 -- 172.21.15.39:0/1000481893 shutdown_connections 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.035+0000 7f0684be3740 1 -- 172.21.15.39:0/1000481893 wait complete. 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 -- start start 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 Processor -- start 2024-09-21T01:16:50.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 0x55b770250e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe79840 0x55b7702513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe8e910 0x55b770251920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f065800e6a0 con 0x55b76fe71660 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f065800fb10 con 0x55b76fe8e910 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f065800ed70 con 0x55b76fe79840 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.036+0000 7f0684be3740 10 client.0 _ll_register_callbacks cb 0x7f0668005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 0x55b770250e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 0x55b770250e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56312/0 (socket says 172.21.15.39:56312) 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 -- 172.21.15.39:0/1287096722 learned_addr learned my addr 172.21.15.39:0/1287096722 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676000640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe79840 0x55b7702513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0677400640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe8e910 0x55b770251920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 -- 172.21.15.39:0/1287096722 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe79840 msgr2=0x55b7702513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe79840 0x55b7702513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 -- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe8e910 msgr2=0x55b770251920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe8e910 0x55b770251920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 -- 172.21.15.39:0/1287096722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b770243a50 con 0x55b76fe71660 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0676a00640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 0x55b770250e80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0668006010 tx=0x7f0668009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9d2a35d20a93dc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.037+0000 7f0660c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0660c00640 1 -- 172.21.15.39:0/1287096722 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668028030 con 0x55b76fe71660 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06700ad640 con 0x55b76fe71660 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0660c00640 1 -- 172.21.15.39:0/1287096722 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f06680189b0 con 0x55b76fe71660 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0660c00640 1 -- 172.21.15.39:0/1287096722 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0668018bf0 con 0x55b76fe71660 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0684be3740 10 client.16083 fetch_fsmap learned FSMap version 40 2024-09-21T01:16:50.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b7702c9830 con 0x55b76fe71660 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.038+0000 7f0660c00640 1 -- 172.21.15.39:0/1287096722 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06700ad640 con 0x55b76fe71660 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0660c00640 1 -- 172.21.15.39:0/1287096722 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f0668003680 con 0x55b76fe71660 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0684be3740 10 client.16083 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0684be3740 10 client.16083 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0684be3740 20 client.16083 populate_metadata read hostname 'smithi039' 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0684be3740 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe9a520 0x55b7702cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0684be3740 4 client.16083 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f65cb898", "release": true, "await": true}] 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f65cb898", "release": true, "await": true}) -- 0x55b7701b5c90 con 0x55b76fe9a520 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.039+0000 7f0676000640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe9a520 0x55b7702cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.040+0000 7f0676000640 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe9a520 0x55b7702cbf50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ce5b8b3e824086 server_cookie=deb06d9127149579 in_seq=0 out_seq=0 2024-09-21T01:16:50.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.040+0000 7f0660c00640 10 client.16083 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:50.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0660c00640 1 -- 172.21.15.39:0/1287096722 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b7701b5c90 con 0x55b76fe9a520 2024-09-21T01:16:50.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0660c00640 10 client.16083 handle_command_reply: tid=0 2024-09-21T01:16:50.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 client.16083 shutdown 2024-09-21T01:16:50.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 msgr2=0x55b770250e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 0x55b770250e80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0668006010 tx=0x7f0668009f00 comp rx=0 tx=0).stop 2024-09-21T01:16:50.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 shutdown_connections 2024-09-21T01:16:50.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55b76fe9a520 0x55b7702cbf50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b76fe8e910 0x55b770251920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55b76fe79840 0x55b7702513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 --2- 172.21.15.39:0/1287096722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b76fe71660 0x55b770250e80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 >> 172.21.15.39:0/1287096722 conn(0x55b77023f0f0 msgr2=0x55b77024cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.048+0000 7f0660c00640 0 client.16083 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 shutdown_connections 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f0684be3740 1 -- 172.21.15.39:0/1287096722 wait complete. 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f0684be3740 20 client.16083 trim_cache size 0 max 16384 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 40, 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:16:50.051 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 52, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "f65cb898": { 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 52, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 137.8, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:16:50.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:16:50.053 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f065c0b6700 msgr2=0x7f065c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f067de00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f065c0b6700 0x7f065c0b8c10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0664031440 tx=0x7f0664005a20 comp rx=0 tx=0).stop 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 msgr2=0x7f06781a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.049+0000 7f067de00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f06781a1f10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f067000efd0 tx=0x7f067000c6a0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 shutdown_connections 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f067810c010 0x7f06781a1f10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f067810b6c0 0x7f067819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f065c0b6700 0x7f065c0b8c10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 --2- 172.21.15.39:0/3453866917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0678105d80 0x7f067819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 >> 172.21.15.39:0/3453866917 conn(0x7f0678101810 msgr2=0x7f0678103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 shutdown_connections 2024-09-21T01:16:50.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:16:50.050+0000 7f067de00640 1 -- 172.21.15.39:0/3453866917 wait complete. 2024-09-21T01:16:50.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:49 smithi150 ceph-mon[34945]: pgmap v802: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 615 KiB/s wr, 27 op/s 2024-09-21T01:16:50.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 527323 3.78 MB/sec execute 557 sec latency 17306.127 ms 2024-09-21T01:16:51.121 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f65cb898', total seconds elapsed: 22.3 2024-09-21T01:16:51.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 529424 3.79 MB/sec execute 558 sec latency 15.171 ms 2024-09-21T01:16:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:51 smithi039 ceph-mon[30415]: pgmap v803: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 623 KiB/s wr, 24 op/s 2024-09-21T01:16:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:51 smithi150 ceph-mon[34945]: pgmap v803: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 623 KiB/s wr, 24 op/s 2024-09-21T01:16:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:51 smithi160 ceph-mon[34696]: pgmap v803: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 623 KiB/s wr, 24 op/s 2024-09-21T01:16:52.201 INFO:tasks.workunit.client.0.smithi039.stdout: 1 530344 3.79 MB/sec execute 559 sec latency 18.927 ms 2024-09-21T01:16:53.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 532239 3.80 MB/sec execute 560 sec latency 15.672 ms 2024-09-21T01:16:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:53 smithi039 ceph-mon[30415]: pgmap v804: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 695 KiB/s wr, 28 op/s 2024-09-21T01:16:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:53 smithi150 ceph-mon[34945]: pgmap v804: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 695 KiB/s wr, 28 op/s 2024-09-21T01:16:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:53 smithi160 ceph-mon[34696]: pgmap v804: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 695 KiB/s wr, 28 op/s 2024-09-21T01:16:54.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 533362 3.81 MB/sec execute 561 sec latency 17.298 ms 2024-09-21T01:16:55.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 534546 3.81 MB/sec execute 562 sec latency 24.151 ms 2024-09-21T01:16:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:55 smithi039 ceph-mon[30415]: pgmap v805: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 805 KiB/s wr, 47 op/s 2024-09-21T01:16:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:55 smithi150 ceph-mon[34945]: pgmap v805: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 805 KiB/s wr, 47 op/s 2024-09-21T01:16:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:55 smithi160 ceph-mon[34696]: pgmap v805: 129 pgs: 129 active+clean; 437 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 805 KiB/s wr, 47 op/s 2024-09-21T01:16:56.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 536591 3.83 MB/sec execute 563 sec latency 17.536 ms 2024-09-21T01:16:57.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 537490 3.83 MB/sec execute 564 sec latency 18.345 ms 2024-09-21T01:16:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:57 smithi039 ceph-mon[30415]: pgmap v806: 129 pgs: 129 active+clean; 439 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 930 KiB/s wr, 56 op/s 2024-09-21T01:16:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:57 smithi150 ceph-mon[34945]: pgmap v806: 129 pgs: 129 active+clean; 439 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 930 KiB/s wr, 56 op/s 2024-09-21T01:16:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:57 smithi160 ceph-mon[34696]: pgmap v806: 129 pgs: 129 active+clean; 439 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 930 KiB/s wr, 56 op/s 2024-09-21T01:16:58.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 538815 3.83 MB/sec execute 565 sec latency 102.593 ms 2024-09-21T01:16:59.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 540350 3.84 MB/sec execute 566 sec latency 39.301 ms 2024-09-21T01:16:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:16:59 smithi039 ceph-mon[30415]: pgmap v807: 129 pgs: 129 active+clean; 448 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 123 op/s 2024-09-21T01:17:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:16:59 smithi150 ceph-mon[34945]: pgmap v807: 129 pgs: 129 active+clean; 448 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 123 op/s 2024-09-21T01:17:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:16:59 smithi160 ceph-mon[34696]: pgmap v807: 129 pgs: 129 active+clean; 448 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 123 op/s 2024-09-21T01:17:00.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 541475 3.85 MB/sec execute 567 sec latency 17.143 ms 2024-09-21T01:17:01.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 543213 3.86 MB/sec execute 568 sec latency 94.544 ms 2024-09-21T01:17:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:01 smithi039 ceph-mon[30415]: pgmap v808: 129 pgs: 129 active+clean; 451 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:17:02.124 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:01 smithi160 ceph-mon[34696]: pgmap v808: 129 pgs: 129 active+clean; 451 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:17:02.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:01 smithi150 ceph-mon[34945]: pgmap v808: 129 pgs: 129 active+clean; 451 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:17:02.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 543585 3.86 MB/sec execute 569 sec latency 87.587 ms 2024-09-21T01:17:03.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 544377 3.85 MB/sec execute 570 sec latency 16.415 ms 2024-09-21T01:17:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:03 smithi039 ceph-mon[30415]: pgmap v809: 129 pgs: 129 active+clean; 452 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-09-21T01:17:04.068 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:03 smithi150 ceph-mon[34945]: pgmap v809: 129 pgs: 129 active+clean; 452 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-09-21T01:17:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:03 smithi160 ceph-mon[34696]: pgmap v809: 129 pgs: 129 active+clean; 452 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-09-21T01:17:04.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 545887 3.86 MB/sec execute 571 sec latency 16.084 ms 2024-09-21T01:17:05.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 547407 3.87 MB/sec execute 572 sec latency 20.702 ms 2024-09-21T01:17:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:05 smithi039 ceph-mon[30415]: pgmap v810: 129 pgs: 129 active+clean; 455 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-21T01:17:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:05 smithi039 ceph-mon[30415]: osdmap e105: 12 total, 12 up, 12 in 2024-09-21T01:17:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:05 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:05 smithi150 ceph-mon[34945]: pgmap v810: 129 pgs: 129 active+clean; 455 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-21T01:17:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:05 smithi150 ceph-mon[34945]: osdmap e105: 12 total, 12 up, 12 in 2024-09-21T01:17:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:05 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:05 smithi160 ceph-mon[34696]: pgmap v810: 129 pgs: 129 active+clean; 455 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-21T01:17:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:05 smithi160 ceph-mon[34696]: osdmap e105: 12 total, 12 up, 12 in 2024-09-21T01:17:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:05 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:06.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 548558 3.87 MB/sec execute 573 sec latency 25.927 ms 2024-09-21T01:17:07.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 550620 3.89 MB/sec execute 574 sec latency 16.570 ms 2024-09-21T01:17:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:07 smithi039 ceph-mon[30415]: pgmap v812: 129 pgs: 129 active+clean; 457 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.2 MiB/s wr, 222 op/s 2024-09-21T01:17:08.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:07 smithi150 ceph-mon[34945]: pgmap v812: 129 pgs: 129 active+clean; 457 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.2 MiB/s wr, 222 op/s 2024-09-21T01:17:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:07 smithi160 ceph-mon[34696]: pgmap v812: 129 pgs: 129 active+clean; 457 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.2 MiB/s wr, 222 op/s 2024-09-21T01:17:08.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 551382 3.89 MB/sec execute 575 sec latency 21.089 ms 2024-09-21T01:17:09.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 552874 3.89 MB/sec execute 576 sec latency 25.034 ms 2024-09-21T01:17:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:09 smithi039 ceph-mon[30415]: pgmap v813: 129 pgs: 129 active+clean; 463 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.0 MiB/s wr, 222 op/s 2024-09-21T01:17:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:09 smithi150 ceph-mon[34945]: pgmap v813: 129 pgs: 129 active+clean; 463 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.0 MiB/s wr, 222 op/s 2024-09-21T01:17:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:09 smithi160 ceph-mon[34696]: pgmap v813: 129 pgs: 129 active+clean; 463 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.0 MiB/s wr, 222 op/s 2024-09-21T01:17:10.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 554541 3.90 MB/sec execute 577 sec latency 11.996 ms 2024-09-21T01:17:11.202 INFO:tasks.workunit.client.0.smithi039.stdout: 1 555595 3.91 MB/sec execute 578 sec latency 16.964 ms 2024-09-21T01:17:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:11 smithi039 ceph-mon[30415]: pgmap v814: 129 pgs: 129 active+clean; 463 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-09-21T01:17:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:11 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:17:12.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:11 smithi150 ceph-mon[34945]: pgmap v814: 129 pgs: 129 active+clean; 463 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-09-21T01:17:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:11 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:17:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:11 smithi160 ceph-mon[34696]: pgmap v814: 129 pgs: 129 active+clean; 463 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-09-21T01:17:12.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:11 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:17:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 557371 3.91 MB/sec execute 579 sec latency 15.721 ms 2024-09-21T01:17:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:17:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:17:12.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:17:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:17:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:17:13.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:17:13.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:17:13.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:17:13.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:17:13.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 558427 3.92 MB/sec execute 580 sec latency 14.840 ms 2024-09-21T01:17:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:13 smithi039 ceph-mon[30415]: pgmap v815: 129 pgs: 129 active+clean; 468 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-09-21T01:17:14.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:13 smithi150 ceph-mon[34945]: pgmap v815: 129 pgs: 129 active+clean; 468 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-09-21T01:17:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:13 smithi160 ceph-mon[34696]: pgmap v815: 129 pgs: 129 active+clean; 468 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-09-21T01:17:14.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 559801 3.92 MB/sec execute 581 sec latency 20.070 ms 2024-09-21T01:17:15.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 561552 3.94 MB/sec execute 582 sec latency 21.903 ms 2024-09-21T01:17:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:15 smithi150 ceph-mon[34945]: pgmap v816: 129 pgs: 129 active+clean; 470 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 181 op/s 2024-09-21T01:17:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:15 smithi150 ceph-mon[34945]: osdmap e106: 12 total, 12 up, 12 in 2024-09-21T01:17:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:15 smithi160 ceph-mon[34696]: pgmap v816: 129 pgs: 129 active+clean; 470 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 181 op/s 2024-09-21T01:17:15.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:15 smithi160 ceph-mon[34696]: osdmap e106: 12 total, 12 up, 12 in 2024-09-21T01:17:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:15 smithi039 ceph-mon[30415]: pgmap v816: 129 pgs: 129 active+clean; 470 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 181 op/s 2024-09-21T01:17:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:15 smithi039 ceph-mon[30415]: osdmap e106: 12 total, 12 up, 12 in 2024-09-21T01:17:16.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 562492 3.93 MB/sec execute 583 sec latency 22.609 ms 2024-09-21T01:17:17.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 564011 3.94 MB/sec execute 584 sec latency 202.711 ms 2024-09-21T01:17:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:17 smithi150 ceph-mon[34945]: pgmap v818: 129 pgs: 129 active+clean; 475 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 197 op/s 2024-09-21T01:17:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:17 smithi160 ceph-mon[34696]: pgmap v818: 129 pgs: 129 active+clean; 475 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 197 op/s 2024-09-21T01:17:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:17 smithi039 ceph-mon[30415]: pgmap v818: 129 pgs: 129 active+clean; 475 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 197 op/s 2024-09-21T01:17:18.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 564011 3.94 MB/sec execute 585 sec latency 1202.857 ms 2024-09-21T01:17:19.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 564011 3.93 MB/sec execute 586 sec latency 2202.979 ms 2024-09-21T01:17:19.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:19 smithi150 ceph-mon[34945]: pgmap v819: 129 pgs: 129 active+clean; 477 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 152 op/s 2024-09-21T01:17:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:19 smithi160 ceph-mon[34696]: pgmap v819: 129 pgs: 129 active+clean; 477 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 152 op/s 2024-09-21T01:17:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:19 smithi039 ceph-mon[30415]: pgmap v819: 129 pgs: 129 active+clean; 477 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 152 op/s 2024-09-21T01:17:20.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 564011 3.92 MB/sec execute 587 sec latency 3203.122 ms 2024-09-21T01:17:21.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 564011 3.91 MB/sec execute 588 sec latency 4203.208 ms 2024-09-21T01:17:21.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:21 smithi039 ceph-mon[30415]: pgmap v820: 129 pgs: 129 active+clean; 481 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 168 op/s 2024-09-21T01:17:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:21 smithi150 ceph-mon[34945]: pgmap v820: 129 pgs: 129 active+clean; 481 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 168 op/s 2024-09-21T01:17:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:21 smithi160 ceph-mon[34696]: pgmap v820: 129 pgs: 129 active+clean; 481 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 168 op/s 2024-09-21T01:17:22.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 564241 3.91 MB/sec execute 589 sec latency 5019.300 ms 2024-09-21T01:17:23.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 565327 3.92 MB/sec execute 590 sec latency 21.396 ms 2024-09-21T01:17:23.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:23 smithi039 ceph-mon[30415]: pgmap v821: 129 pgs: 129 active+clean; 484 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 136 op/s 2024-09-21T01:17:23.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:23 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:23.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:23 smithi150 ceph-mon[34945]: pgmap v821: 129 pgs: 129 active+clean; 484 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 136 op/s 2024-09-21T01:17:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:23 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:23 smithi160 ceph-mon[34696]: pgmap v821: 129 pgs: 129 active+clean; 484 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 136 op/s 2024-09-21T01:17:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:23 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:24.203 INFO:tasks.workunit.client.0.smithi039.stdout: 1 566655 3.92 MB/sec execute 591 sec latency 21.590 ms 2024-09-21T01:17:25.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 568006 3.92 MB/sec execute 592 sec latency 193.077 ms 2024-09-21T01:17:25.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:25 smithi150 ceph-mon[34945]: pgmap v822: 129 pgs: 129 active+clean; 484 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 108 op/s 2024-09-21T01:17:25.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:25 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:25 smithi160 ceph-mon[34696]: pgmap v822: 129 pgs: 129 active+clean; 484 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 108 op/s 2024-09-21T01:17:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:25 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:25 smithi039 ceph-mon[30415]: pgmap v822: 129 pgs: 129 active+clean; 484 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 108 op/s 2024-09-21T01:17:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:25 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:26.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 569012 3.93 MB/sec execute 593 sec latency 15.963 ms 2024-09-21T01:17:27.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 569055 3.92 MB/sec execute 594 sec latency 853.948 ms 2024-09-21T01:17:27.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:27 smithi039 ceph-mon[30415]: pgmap v823: 129 pgs: 129 active+clean; 487 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T01:17:27.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:27 smithi150 ceph-mon[34945]: pgmap v823: 129 pgs: 129 active+clean; 487 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T01:17:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:27 smithi160 ceph-mon[34696]: pgmap v823: 129 pgs: 129 active+clean; 487 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-09-21T01:17:28.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 569532 3.92 MB/sec execute 595 sec latency 1426.408 ms 2024-09-21T01:17:29.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 571545 3.93 MB/sec execute 596 sec latency 25.548 ms 2024-09-21T01:17:29.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:29 smithi150 ceph-mon[34945]: pgmap v824: 129 pgs: 129 active+clean; 491 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-09-21T01:17:29.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:29 smithi160 ceph-mon[34696]: pgmap v824: 129 pgs: 129 active+clean; 491 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-09-21T01:17:29.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:29 smithi039 ceph-mon[30415]: pgmap v824: 129 pgs: 129 active+clean; 491 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-09-21T01:17:30.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 572465 3.93 MB/sec execute 597 sec latency 19.709 ms 2024-09-21T01:17:31.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 573481 3.94 MB/sec execute 598 sec latency 18.627 ms 2024-09-21T01:17:31.626 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:31 smithi039 ceph-mon[30415]: pgmap v825: 129 pgs: 129 active+clean; 491 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-09-21T01:17:31.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:31 smithi150 ceph-mon[34945]: pgmap v825: 129 pgs: 129 active+clean; 491 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-09-21T01:17:31.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:31 smithi160 ceph-mon[34696]: pgmap v825: 129 pgs: 129 active+clean; 491 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-09-21T01:17:32.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 573714 3.93 MB/sec execute 599 sec latency 811.983 ms 2024-09-21T01:17:33.204 INFO:tasks.workunit.client.0.smithi039.stdout: 1 cleanup 600 sec 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: 0 cleanup 600 sec 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: Operation Count AvgLat MaxLat 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: ---------------------------------------- 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: NTCreateX 75062 2.301 21564.355 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: Close 55149 0.005 0.662 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: Rename 3180 47.046 24310.713 2024-09-21T01:17:33.598 INFO:tasks.workunit.client.0.smithi039.stdout: Unlink 15147 5.535 21704.175 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Deltree 2 83.503 122.480 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Mkdir 1 0.014 0.014 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Qpathinfo 68046 1.266 2298.635 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Qfileinfo 11939 0.004 2.385 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Qfsinfo 12476 0.366 399.442 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Sfileinfo 6120 6.969 24066.848 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Find 26313 0.682 9855.364 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: WriteX 37470 0.190 307.529 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: ReadX 117780 0.056 602.510 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: LockX 246 0.898 4.793 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: UnlockX 246 0.767 2.384 2024-09-21T01:17:33.599 INFO:tasks.workunit.client.0.smithi039.stdout: Flush 5257 5.070 1027.961 2024-09-21T01:17:33.600 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-21T01:17:33.600 INFO:tasks.workunit.client.0.smithi039.stdout:Throughput 3.92941 MB/sec 1 clients 1 procs max_latency=24310.725 ms 2024-09-21T01:17:33.600 INFO:tasks.workunit.client.0.smithi039.stderr:+ dbench 10 2024-09-21T01:17:33.600 INFO:tasks.workunit.client.0.smithi039.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T01:17:33.600 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-21T01:17:33.601 INFO:tasks.workunit.client.0.smithi039.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T01:17:33.602 INFO:tasks.workunit.client.0.smithi039.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-21T01:17:33.702 INFO:tasks.workunit.client.0.smithi039.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-21T01:17:33.702 INFO:tasks.workunit.client.0.smithi039.stdout:releasing clients 2024-09-21T01:17:33.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:33 smithi150 ceph-mon[34945]: pgmap v826: 129 pgs: 129 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-09-21T01:17:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:33 smithi160 ceph-mon[34696]: pgmap v826: 129 pgs: 129 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-09-21T01:17:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:33 smithi039 ceph-mon[30415]: pgmap v826: 129 pgs: 129 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-09-21T01:17:34.703 INFO:tasks.workunit.client.0.smithi039.stdout: 10 387 137.95 MB/sec warmup 1 sec latency 182.437 ms 2024-09-21T01:17:35.703 INFO:tasks.workunit.client.0.smithi039.stdout: 10 884 102.95 MB/sec warmup 2 sec latency 1182.482 ms 2024-09-21T01:17:35.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:35 smithi150 ceph-mon[34945]: pgmap v827: 129 pgs: 129 active+clean; 500 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.4 MiB/s wr, 170 op/s 2024-09-21T01:17:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:35 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:35 smithi160 ceph-mon[34696]: pgmap v827: 129 pgs: 129 active+clean; 500 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.4 MiB/s wr, 170 op/s 2024-09-21T01:17:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:35 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:35 smithi039 ceph-mon[30415]: pgmap v827: 129 pgs: 129 active+clean; 500 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.4 MiB/s wr, 170 op/s 2024-09-21T01:17:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:35 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:36.703 INFO:tasks.workunit.client.0.smithi039.stdout: 10 1326 81.27 MB/sec warmup 3 sec latency 2182.526 ms 2024-09-21T01:17:37.704 INFO:tasks.workunit.client.0.smithi039.stdout: 10 1489 61.27 MB/sec warmup 4 sec latency 3184.153 ms 2024-09-21T01:17:37.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:37 smithi150 ceph-mon[34945]: pgmap v828: 129 pgs: 129 active+clean; 503 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-09-21T01:17:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:37 smithi160 ceph-mon[34696]: pgmap v828: 129 pgs: 129 active+clean; 503 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-09-21T01:17:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:37 smithi039 ceph-mon[30415]: pgmap v828: 129 pgs: 129 active+clean; 503 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-09-21T01:17:38.704 INFO:tasks.workunit.client.0.smithi039.stdout: 10 2302 53.80 MB/sec warmup 5 sec latency 4184.218 ms 2024-09-21T01:17:38.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:38 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:38.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:38 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:39.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:38 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:39 smithi039 ceph-mon[30415]: pgmap v829: 129 pgs: 129 active+clean; 505 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-21T01:17:39.704 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3048 54.73 MB/sec warmup 6 sec latency 5184.274 ms 2024-09-21T01:17:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:39 smithi160 ceph-mon[34696]: pgmap v829: 129 pgs: 129 active+clean; 505 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-21T01:17:40.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:39 smithi150 ceph-mon[34945]: pgmap v829: 129 pgs: 129 active+clean; 505 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-21T01:17:40.704 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3475 52.49 MB/sec warmup 7 sec latency 6184.325 ms 2024-09-21T01:17:41.711 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3653 46.24 MB/sec warmup 8 sec latency 7191.076 ms 2024-09-21T01:17:41.847 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:41 smithi039 ceph-mon[30415]: pgmap v830: 129 pgs: 129 active+clean; 512 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 203 op/s 2024-09-21T01:17:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:41 smithi160 ceph-mon[34696]: pgmap v830: 129 pgs: 129 active+clean; 512 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 203 op/s 2024-09-21T01:17:42.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:41 smithi150 ceph-mon[34945]: pgmap v830: 129 pgs: 129 active+clean; 512 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 203 op/s 2024-09-21T01:17:42.724 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3663 41.05 MB/sec warmup 9 sec latency 8204.014 ms 2024-09-21T01:17:43.724 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3663 36.95 MB/sec warmup 10 sec latency 9204.061 ms 2024-09-21T01:17:43.810 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:43 smithi039 ceph-mon[30415]: pgmap v831: 129 pgs: 129 active+clean; 517 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 209 op/s 2024-09-21T01:17:43.810 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:43 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:43 smithi150 ceph-mon[34945]: pgmap v831: 129 pgs: 129 active+clean; 517 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 209 op/s 2024-09-21T01:17:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:43 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:43 smithi160 ceph-mon[34696]: pgmap v831: 129 pgs: 129 active+clean; 517 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 209 op/s 2024-09-21T01:17:44.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:43 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:44.724 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3663 33.60 MB/sec warmup 11 sec latency 10204.129 ms 2024-09-21T01:17:45.724 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3670 30.81 MB/sec warmup 12 sec latency 11204.192 ms 2024-09-21T01:17:45.822 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:45 smithi039 ceph-mon[30415]: pgmap v832: 129 pgs: 129 active+clean; 565 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.2 MiB/s wr, 231 op/s 2024-09-21T01:17:45.823 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:45 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:45 smithi150 ceph-mon[34945]: pgmap v832: 129 pgs: 129 active+clean; 565 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.2 MiB/s wr, 231 op/s 2024-09-21T01:17:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:45 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:45 smithi160 ceph-mon[34696]: pgmap v832: 129 pgs: 129 active+clean; 565 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.2 MiB/s wr, 231 op/s 2024-09-21T01:17:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:45 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:46.725 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3823 28.70 MB/sec warmup 13 sec latency 11422.906 ms 2024-09-21T01:17:47.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3862 26.76 MB/sec warmup 14 sec latency 776.440 ms 2024-09-21T01:17:47.919 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:47 smithi039 ceph-mon[30415]: pgmap v833: 129 pgs: 129 active+clean; 584 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 236 op/s 2024-09-21T01:17:48.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:47 smithi150 ceph-mon[34945]: pgmap v833: 129 pgs: 129 active+clean; 584 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 236 op/s 2024-09-21T01:17:48.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:47 smithi160 ceph-mon[34696]: pgmap v833: 129 pgs: 129 active+clean; 584 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 236 op/s 2024-09-21T01:17:48.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3864 24.98 MB/sec warmup 15 sec latency 1776.485 ms 2024-09-21T01:17:49.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3864 23.42 MB/sec warmup 16 sec latency 2776.534 ms 2024-09-21T01:17:49.862 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:49 smithi039 ceph-mon[30415]: pgmap v834: 129 pgs: 129 active+clean; 668 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 17 MiB/s wr, 287 op/s 2024-09-21T01:17:50.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:49 smithi150 ceph-mon[34945]: pgmap v834: 129 pgs: 129 active+clean; 668 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 17 MiB/s wr, 287 op/s 2024-09-21T01:17:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:49 smithi160 ceph-mon[34696]: pgmap v834: 129 pgs: 129 active+clean; 668 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 17 MiB/s wr, 287 op/s 2024-09-21T01:17:50.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 3988 22.25 MB/sec warmup 17 sec latency 3505.865 ms 2024-09-21T01:17:51.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 4480 22.90 MB/sec warmup 18 sec latency 140.481 ms 2024-09-21T01:17:51.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:51 smithi150 ceph-mon[34945]: pgmap v835: 129 pgs: 129 active+clean; 682 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 18 MiB/s wr, 263 op/s 2024-09-21T01:17:51.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:51 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:51.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:51 smithi160 ceph-mon[34696]: pgmap v835: 129 pgs: 129 active+clean; 682 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 18 MiB/s wr, 263 op/s 2024-09-21T01:17:51.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:51 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:51 smithi039 ceph-mon[30415]: pgmap v835: 129 pgs: 129 active+clean; 682 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 18 MiB/s wr, 263 op/s 2024-09-21T01:17:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:51 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:52.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 5165 23.03 MB/sec warmup 19 sec latency 157.907 ms 2024-09-21T01:17:53.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 5849 24.03 MB/sec warmup 20 sec latency 49.132 ms 2024-09-21T01:17:53.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:53 smithi150 ceph-mon[34945]: pgmap v836: 129 pgs: 129 active+clean; 726 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 21 MiB/s wr, 268 op/s 2024-09-21T01:17:53.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:53 smithi160 ceph-mon[34696]: pgmap v836: 129 pgs: 129 active+clean; 726 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 21 MiB/s wr, 268 op/s 2024-09-21T01:17:53.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:53 smithi039 ceph-mon[30415]: pgmap v836: 129 pgs: 129 active+clean; 726 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 21 MiB/s wr, 268 op/s 2024-09-21T01:17:54.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 6508 25.43 MB/sec warmup 21 sec latency 156.261 ms 2024-09-21T01:17:55.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 6818 25.28 MB/sec warmup 22 sec latency 144.768 ms 2024-09-21T01:17:55.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:55 smithi150 ceph-mon[34945]: pgmap v837: 129 pgs: 129 active+clean; 719 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 22 MiB/s wr, 318 op/s 2024-09-21T01:17:55.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:55 smithi160 ceph-mon[34696]: pgmap v837: 129 pgs: 129 active+clean; 719 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 22 MiB/s wr, 318 op/s 2024-09-21T01:17:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:55 smithi039 ceph-mon[30415]: pgmap v837: 129 pgs: 129 active+clean; 719 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 22 MiB/s wr, 318 op/s 2024-09-21T01:17:56.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:56 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:56.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 7187 24.41 MB/sec warmup 23 sec latency 131.917 ms 2024-09-21T01:17:56.816 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:56 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:56.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:56 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:17:57.236 INFO:tasks.quiescer.fs.[cephfs]:Total duration (37.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:17:57.237 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.85 2024-09-21T01:17:57.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:17:57.731 INFO:tasks.workunit.client.0.smithi039.stdout: 10 7900 24.49 MB/sec warmup 24 sec latency 71.795 ms 2024-09-21T01:17:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:57 smithi150 ceph-mon[34945]: pgmap v838: 129 pgs: 129 active+clean; 723 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 301 op/s 2024-09-21T01:17:57.849 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:17:57.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:57 smithi160 ceph-mon[34696]: pgmap v838: 129 pgs: 129 active+clean; 723 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 301 op/s 2024-09-21T01:17:57.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:57 smithi039 ceph-mon[30415]: pgmap v838: 129 pgs: 129 active+clean; 723 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 301 op/s 2024-09-21T01:17:58.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 -- 172.21.15.39:0/3759680114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2484109e10 msgr2=0x7f2484110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:17:58.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 --2- 172.21.15.39:0/3759680114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2484109e10 0x7f2484110640 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f247c009ca0 tx=0x7f247c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 -- 172.21.15.39:0/3759680114 shutdown_connections 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 --2- 172.21.15.39:0/3759680114 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2484109e10 0x7f2484110640 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 --2- 172.21.15.39:0/3759680114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24841094c0 0x7f24841098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 --2- 172.21.15.39:0/3759680114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2484103b80 0x7f2484103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 -- 172.21.15.39:0/3759680114 >> 172.21.15.39:0/3759680114 conn(0x7f24840ff590 msgr2=0x7f24841019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 -- 172.21.15.39:0/3759680114 shutdown_connections 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.575+0000 7f248ba00640 1 -- 172.21.15.39:0/3759680114 wait complete. 2024-09-21T01:17:58.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.577+0000 7f248ba00640 1 Processor -- start 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.577+0000 7f248ba00640 1 -- start start 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f248ba00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2484103b80 0x7f2484198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f248ba00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 0x7f2484199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f248ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2484109e10 0x7f2484199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f248ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2484113b20 con 0x7f2484109e10 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f248ba00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24841139a0 con 0x7f2484103b80 2024-09-21T01:17:58.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f248ba00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2484113ca0 con 0x7f24841094c0 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f2483e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 0x7f2484199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f2483e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 0x7f2484199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53060/0 (socket says 172.21.15.39:53060) 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f2483e00640 1 -- 172.21.15.39:0/1861455932 learned_addr learned my addr 172.21.15.39:0/1861455932 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f2489000640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2484103b80 0x7f2484198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.578+0000 7f2489a00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2484109e10 0x7f2484199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2483e00640 1 -- 172.21.15.39:0/1861455932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2484103b80 msgr2=0x7f2484198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2483e00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2484103b80 0x7f2484198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2483e00640 1 -- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2484109e10 msgr2=0x7f2484199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:17:58.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2483e00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2484109e10 0x7f2484199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2483e00640 1 -- 172.21.15.39:0/1861455932 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24841a0270 con 0x7f24841094c0 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2483e00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 0x7f2484199510 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f247000e970 tx=0x7f247000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247000cd80 con 0x7f24841094c0 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.579+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24841a0560 con 0x7f24841094c0 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.580+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24700035f0 con 0x7f24841094c0 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.580+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2470010690 con 0x7f24841094c0 2024-09-21T01:17:58.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.580+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24841a0af0 con 0x7f24841094c0 2024-09-21T01:17:58.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.581+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2484103f90 con 0x7f24841094c0 2024-09-21T01:17:58.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.584+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2470003aa0 con 0x7f24841094c0 2024-09-21T01:17:58.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.586+0000 7f2481600640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f246c0b6780 0x7f246c0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:17:58.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.586+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9511+0+0 (secure 0 0 0) 0x7f24700e7ed0 con 0x7f24841094c0 2024-09-21T01:17:58.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.587+0000 7f2489000640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f246c0b6780 0x7f246c0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:17:58.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.587+0000 7f2489000640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f246c0b6780 0x7f246c0b8c90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2484110420 tx=0x7f2478005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:17:58.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.589+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f24700adfe0 con 0x7f24841094c0 2024-09-21T01:17:58.731 INFO:tasks.workunit.client.0.smithi039.stdout: 10 8885 26.24 MB/sec warmup 25 sec latency 69.360 ms 2024-09-21T01:17:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.938+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2484100b20 con 0x7f24841094c0 2024-09-21T01:17:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.939+0000 7f2481600640 1 -- 172.21.15.39:0/1861455932 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11916 (secure 0 0 0) 0x7f24700b23c0 con 0x7f24841094c0 2024-09-21T01:17:58.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:17:58.939 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":49,"btime":"2024-09-21T01:17:55:505807+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:17:54.531815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:17:58.940 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 49 2024-09-21T01:17:58.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f246c0b6780 msgr2=0x7f246c0b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f246c0b6780 0x7f246c0b8c90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2484110420 tx=0x7f2478005f90 comp rx=0 tx=0).stop 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 msgr2=0x7f2484199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 0x7f2484199510 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f247000e970 tx=0x7f247000ee30 comp rx=0 tx=0).stop 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 shutdown_connections 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2484109e10 0x7f2484199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f24841094c0 0x7f2484199510 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f247000e970 tx=0x7f247000ee30 comp rx=0 tx=0).stop 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f246c0b6780 0x7f246c0b8c90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 --2- 172.21.15.39:0/1861455932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2484103b80 0x7f2484198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:17:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.942+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 >> 172.21.15.39:0/1861455932 conn(0x7f24840ff590 msgr2=0x7f2484108a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:17:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.944+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 shutdown_connections 2024-09-21T01:17:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:17:58.944+0000 7f248ba00640 1 -- 172.21.15.39:0/1861455932 wait complete. 2024-09-21T01:17:59.660 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:59 smithi039 ceph-mon[30415]: pgmap v839: 129 pgs: 129 active+clean; 730 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 320 op/s 2024-09-21T01:17:59.661 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:17:59 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1861455932' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:17:59.730 INFO:tasks.workunit.client.0.smithi039.stdout: 10 9563 26.15 MB/sec warmup 26 sec latency 143.194 ms 2024-09-21T01:17:59.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:59 smithi150 ceph-mon[34945]: pgmap v839: 129 pgs: 129 active+clean; 730 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 320 op/s 2024-09-21T01:17:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:17:59 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1861455932' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:17:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:59 smithi160 ceph-mon[34696]: pgmap v839: 129 pgs: 129 active+clean; 730 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 320 op/s 2024-09-21T01:17:59.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:17:59 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1861455932' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:00.332 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0' 2024-09-21T01:18:00.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0 2024-09-21T01:18:00.732 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10427 27.41 MB/sec warmup 27 sec latency 117.277 ms 2024-09-21T01:18:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:01.694 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:01 smithi039 ceph-mon[30415]: pgmap v840: 129 pgs: 129 active+clean; 745 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.5 MiB/s wr, 329 op/s 2024-09-21T01:18:01.694 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:01 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:18:01.739 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10894 27.09 MB/sec warmup 28 sec latency 296.447 ms 2024-09-21T01:18:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:01 smithi150 ceph-mon[34945]: pgmap v840: 129 pgs: 129 active+clean; 745 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.5 MiB/s wr, 329 op/s 2024-09-21T01:18:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:01 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:18:01.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:01 smithi160 ceph-mon[34696]: pgmap v840: 129 pgs: 129 active+clean; 745 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.5 MiB/s wr, 329 op/s 2024-09-21T01:18:01.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:01 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:18:02.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.809+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2651071539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 msgr2=0x7f4fb8107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.809+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2651071539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 0x7f4fb8107f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009cd0 tx=0x7f4fb002a8f0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2651071539 shutdown_connections 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2651071539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 0x7f4fb8107f00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2651071539 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb8107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2651071539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fb81025e0 0x7f4fb81029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2651071539 >> 172.21.15.39:0/2651071539 conn(0x7f4fb80fe330 msgr2=0x7f4fb8100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2651071539 shutdown_connections 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.810+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2651071539 wait complete. 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.811+0000 7f4fb7e00640 1 Processor -- start 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.812+0000 7f4fb7e00640 1 -- start start 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.812+0000 7f4fb7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fb81025e0 0x7f4fb806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.813+0000 7f4fb7e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb806d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.813+0000 7f4fb7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 0x7f4fb8072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.813+0000 7f4fb6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fb81025e0 0x7f4fb806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb8119f10 con 0x7f4fb81025e0 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb8119d90 con 0x7f4fb8107a20 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb7e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb811a090 con 0x7f4fb8102f30 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb6000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb806d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb6000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb806d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53084/0 (socket says 172.21.15.39:53084) 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb6000640 1 -- 172.21.15.39:0/2311807073 learned_addr learned my addr 172.21.15.39:0/2311807073 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.814+0000 7f4fb7400640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 0x7f4fb8072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.815+0000 7f4fb6000640 1 -- 172.21.15.39:0/2311807073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 msgr2=0x7f4fb8072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.815+0000 7f4fb6000640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 0x7f4fb8072930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.815+0000 7f4fb6000640 1 -- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fb81025e0 msgr2=0x7f4fb806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.815+0000 7f4fb6000640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fb81025e0 0x7f4fb806d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.815+0000 7f4fb6000640 1 -- 172.21.15.39:0/2311807073 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb8072f70 con 0x7f4fb8102f30 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.815+0000 7f4fb6000640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb806d670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400e9f0 tx=0x7f4fa400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.816+0000 7f4faf400640 1 -- 172.21.15.39:0/2311807073 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400ccc0 con 0x7f4fb8102f30 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.816+0000 7f4faf400640 1 -- 172.21.15.39:0/2311807073 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fa40108e0 con 0x7f4fb8102f30 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.816+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb8073260 con 0x7f4fb8102f30 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.817+0000 7f4faf400640 1 -- 172.21.15.39:0/2311807073 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4018860 con 0x7f4fb8102f30 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.817+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb8073aa0 con 0x7f4fb8102f30 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.826+0000 7f4fc46bf740 1 Processor -- start 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.826+0000 7f4fc46bf740 1 -- start start 2024-09-21T01:18:02.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b60d850 0x55e24b9b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 0x55e24b9b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e24b92aa90 con 0x55e24b5e4990 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e24b919ec0 con 0x55e24b60d850 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e24b91dcd0 con 0x55e24b5ecb70 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fb6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 0x55e24b9b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.827+0000 7f4fb6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 0x55e24b9b50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39946/0 (socket says 172.21.15.39:39946) 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.828+0000 7f4fb6a00640 1 -- 172.21.15.39:0/1888302907 learned_addr learned my addr 172.21.15.39:0/1888302907 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.828+0000 7f4fb7400640 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b60d850 0x55e24b9b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.829+0000 7f4faf400640 1 -- 172.21.15.39:0/2311807073 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4fa4010430 con 0x7f4fb8102f30 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.829+0000 7f4fb6a00640 1 -- 172.21.15.39:0/1888302907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 msgr2=0x55e24b9b5610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:18:02.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.829+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9b5610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.829+0000 7f4fb6a00640 1 -- 172.21.15.39:0/1888302907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b60d850 msgr2=0x55e24b9b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.829+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b60d850 0x55e24b9b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.829+0000 7f4fb6a00640 1 -- 172.21.15.39:0/1888302907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24b8bf9d0 con 0x55e24b5e4990 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.830+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 0x55e24b9b50d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800cc20 tx=0x7f4fa800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15255e823c5f443 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.831+0000 7f4fa3e00640 1 -- 172.21.15.39:0/1888302907 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8014070 con 0x55e24b5e4990 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.831+0000 7f4fa3e00640 1 -- 172.21.15.39:0/1888302907 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fa80047a0 con 0x55e24b5e4990 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.831+0000 7f4fa3e00640 1 -- 172.21.15.39:0/1888302907 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8010650 con 0x55e24b5e4990 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.832+0000 7f4faf400640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f9c0b66f0 0x7f4f9c0b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.833+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1888302907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 msgr2=0x55e24b9b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.833+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 0x55e24b9b50d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800cc20 tx=0x7f4fa800e720 comp rx=0 tx=0).stop 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.833+0000 7f4faf400640 1 -- 172.21.15.39:0/2311807073 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9511+0+0 (secure 0 0 0) 0x7f4fa40e7eb0 con 0x7f4fb8102f30 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.834+0000 7f4fb7400640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f9c0b66f0 0x7f4f9c0b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1888302907 shutdown_connections 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5e4990 0x55e24b9b50d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1888302907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b60d850 0x55e24b9b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1888302907 >> 172.21.15.39:0/1888302907 conn(0x55e24b601c40 msgr2=0x55e24b9b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1888302907 shutdown_connections 2024-09-21T01:18:02.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.835+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1888302907 wait complete. 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.836+0000 7f4fc46bf740 1 -- start start 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.836+0000 7f4fb7400640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f9c0b66f0 0x7f4f9c0b8c00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0004af0 tx=0x7f4fb0005d80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.836+0000 7f4fc46bf740 1 Processor -- start 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9c06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5ecb70 0x55e24b9c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b60d850 0x55e24b9dd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e24b77b7b0 con 0x55e24b5ecb70 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e24b8be650 con 0x55e24b5e4990 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e24b9d4380 con 0x55e24b60d850 2024-09-21T01:18:02.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fc46bf740 10 client.0 _ll_register_callbacks cb 0x55e24b6250a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fb6a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9c06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fb6a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9c06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:34616/0 (socket says 172.21.15.39:34616) 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fb6a00640 1 -- 172.21.15.39:0/4105818944 learned_addr learned my addr 172.21.15.39:0/4105818944 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fb7400640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b60d850 0x55e24b9dd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.840+0000 7f4fb6000640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5ecb70 0x55e24b9c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6a00640 1 -- 172.21.15.39:0/4105818944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b60d850 msgr2=0x55e24b9dd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b60d850 0x55e24b9dd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6a00640 1 -- 172.21.15.39:0/4105818944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5ecb70 msgr2=0x55e24b9c0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5ecb70 0x55e24b9c0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6a00640 1 -- 172.21.15.39:0/4105818944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24b687ba0 con 0x55e24b5e4990 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6000640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5ecb70 0x55e24b9c0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9c06c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8005da0 tx=0x7f4fa800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dd175844db02ca2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:02.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fa0c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fa0c00640 1 -- 172.21.15.39:0/4105818944 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8014030 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.841+0000 7f4fa0c00640 1 -- 172.21.15.39:0/4105818944 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9511+0+0 (secure 0 0 0) 0x7f4fa8004260 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.843+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fa80047a0 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.843+0000 7f4fa0c00640 1 -- 172.21.15.39:0/4105818944 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f4fa80044a0 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.843+0000 7f4fc46bf740 10 client.26126 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.843+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55e24b9e02a0 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.844+0000 7f4fa0c00640 1 -- 172.21.15.39:0/4105818944 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fa80047a0 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fa0c00640 1 -- 172.21.15.39:0/4105818944 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7f4fa8010430 con 0x55e24b5e4990 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fc46bf740 10 client.26126 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:02.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fc46bf740 10 client.26126 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fc46bf740 20 client.26126 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b5cc1d0 0x55e24b9e36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fc46bf740 4 client.26126 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.845+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e24b92c9e0 con 0x55e24b5cc1d0 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.846+0000 7f4fb6000640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b5cc1d0 0x55e24b9e36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.848+0000 7f4fa0c00640 10 client.26126 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.848+0000 7f4fb6000640 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b5cc1d0 0x55e24b9e36b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e879ed819924d8 server_cookie=1e60f4f1655ec31f in_seq=0 out_seq=0 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fa0c00640 1 -- 172.21.15.39:0/4105818944 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e24b92c9e0 con 0x55e24b5cc1d0 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fa0c00640 10 client.26126 handle_command_reply: tid=0 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 client.26126 shutdown 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 msgr2=0x55e24b9c06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9c06c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8005da0 tx=0x7f4fa800c960 comp rx=0 tx=0).stop 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 shutdown_connections 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b5cc1d0 0x55e24b9e36b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fa0c00640 0 client.26126 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b60d850 0x55e24b9dd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5ecb70 0x55e24b9c0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/4105818944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9c06c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.849+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 >> 172.21.15.39:0/4105818944 conn(0x55e24b601c40 msgr2=0x55e24b9b7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.850+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 shutdown_connections 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.850+0000 7f4fc46bf740 1 -- 172.21.15.39:0/4105818944 wait complete. 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.850+0000 7f4fc46bf740 20 client.26126 trim_cache size 0 max 16384 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.886+0000 7f4fc46bf740 1 Processor -- start 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.886+0000 7f4fc46bf740 1 -- start start 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.886+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.886+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 0x55e24b9dd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5ecb70 0x55e24b9ddab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb81b96b0 con 0x55e24b5cc1d0 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb81b9de0 con 0x55e24b5ecb70 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb81b7640 con 0x55e24b5e4990 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fb6000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 0x55e24b9dd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fb6000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 0x55e24b9dd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53108/0 (socket says 172.21.15.39:53108) 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fb6000640 1 -- 172.21.15.39:0/3745066632 learned_addr learned my addr 172.21.15.39:0/3745066632 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fb7400640 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5ecb70 0x55e24b9ddab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.887+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.888+0000 7f4fb6000640 1 -- 172.21.15.39:0/3745066632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5ecb70 msgr2=0x55e24b9ddab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.888+0000 7f4fb6000640 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5ecb70 0x55e24b9ddab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.888+0000 7f4fb6000640 1 -- 172.21.15.39:0/3745066632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 msgr2=0x55e24b9c20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.888+0000 7f4fb6000640 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9c20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.888+0000 7f4fb6000640 1 -- 172.21.15.39:0/3745066632 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24b9ae130 con 0x55e24b5e4990 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.888+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9c20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:18:02.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.889+0000 7f4fb6000640 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 0x55e24b9dd570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40ad850 tx=0x7f4fa40096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda99cc77e452e16 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.889+0000 7f4fa3400640 1 -- 172.21.15.39:0/3745066632 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40adfc0 con 0x55e24b5e4990 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.890+0000 7f4fa3400640 1 -- 172.21.15.39:0/3745066632 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e24b92c9e0 con 0x55e24b5e4990 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.890+0000 7f4fa3400640 1 -- 172.21.15.39:0/3745066632 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40e5440 con 0x55e24b5e4990 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.891+0000 7f4fc46bf740 1 -- 172.21.15.39:0/3745066632 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 msgr2=0x55e24b9dd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.891+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 0x55e24b9dd570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40ad850 tx=0x7f4fa40096a0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 -- 172.21.15.39:0/3745066632 shutdown_connections 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5ecb70 0x55e24b9ddab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5e4990 0x55e24b9dd570 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/3745066632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9c20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 -- 172.21.15.39:0/3745066632 >> 172.21.15.39:0/3745066632 conn(0x55e24b601c40 msgr2=0x55e24b9c19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 -- 172.21.15.39:0/3745066632 shutdown_connections 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.892+0000 7f4fc46bf740 1 -- 172.21.15.39:0/3745066632 wait complete. 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.893+0000 7f4fc46bf740 1 -- start start 2024-09-21T01:18:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.893+0000 7f4fc46bf740 1 Processor -- start 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9d9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9d4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb8075410 con 0x55e24b5cc1d0 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb8111d20 con 0x55e24b5e4990 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb8110f80 con 0x55e24b5ecb70 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fc46bf740 10 client.0 _ll_register_callbacks cb 0x7f4fa800ca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.894+0000 7f4fb7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9d4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb7400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9d4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53120/0 (socket says 172.21.15.39:53120) 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb7400640 1 -- 172.21.15.39:0/1491317675 learned_addr learned my addr 172.21.15.39:0/1491317675 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb6000640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9d9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb7400640 1 -- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 msgr2=0x55e24b9d9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb7400640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9d9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.895+0000 7f4fb7400640 1 -- 172.21.15.39:0/1491317675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 msgr2=0x55e24b9d97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.896+0000 7f4fb7400640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9d97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.896+0000 7f4fb7400640 1 -- 172.21.15.39:0/1491317675 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24b9d4ad0 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.896+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fb7400640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9d4590 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009cd0 tx=0x7f4fb00095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=704a195ae066a0bb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fa0c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fa0c00640 1 -- 172.21.15.39:0/1491317675 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0059060 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e24b92c9e0 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fa0c00640 1 -- 172.21.15.39:0/1491317675 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9511+0+0 (secure 0 0 0) 0x7f4fb0003d80 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fa0c00640 1 -- 172.21.15.39:0/1491317675 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f4fb0060050 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.898+0000 7f4fc46bf740 10 client.26122 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.899+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55e24ba4e6d0 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.899+0000 7f4fa0c00640 1 -- 172.21.15.39:0/1491317675 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e24b92c9e0 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fa0c00640 1 -- 172.21.15.39:0/1491317675 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7f4fb0054750 con 0x55e24b5ecb70 2024-09-21T01:18:02.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fc46bf740 10 client.26122 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fc46bf740 10 client.26122 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fc46bf740 20 client.26122 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b60d850 0x55e24ba50f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fc46bf740 4 client.26122 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}] 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}) -- 0x55e24b928490 con 0x55e24b60d850 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.901+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b60d850 0x55e24ba50f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.902+0000 7f4fb6a00640 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b60d850 0x55e24ba50f10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42f570f6f3c21608 server_cookie=76d6afac35f5cf82 in_seq=0 out_seq=0 2024-09-21T01:18:02.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:01.902+0000 7f4fa0c00640 10 client.26122 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:02.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.462+0000 7f4fa0c00640 1 -- 172.21.15.39:0/1491317675 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e24b928490 con 0x55e24b60d850 2024-09-21T01:18:02.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.462+0000 7f4fa0c00640 10 client.26122 handle_command_reply: tid=0 2024-09-21T01:18:02.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.462+0000 7f4fc46bf740 1 client.26122 shutdown 2024-09-21T01:18:02.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.462+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 msgr2=0x55e24b9d4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.462+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9d4590 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009cd0 tx=0x7f4fb00095c0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 shutdown_connections 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e24b60d850 0x55e24ba50f10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e24b5ecb70 0x55e24b9d4590 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e24b5e4990 0x55e24b9d9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 --2- 172.21.15.39:0/1491317675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e24b5cc1d0 0x55e24b9d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 >> 172.21.15.39:0/1491317675 conn(0x55e24b601c40 msgr2=0x55e24b9b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fa0c00640 0 client.26122 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 shutdown_connections 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 1 -- 172.21.15.39:0/1491317675 wait complete. 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.463+0000 7f4fc46bf740 20 client.26122 trim_cache size 0 max 16384 2024-09-21T01:18:02.464 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 50, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 54, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "b8d5ab3f": { 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 54, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:18:02.465 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 138.8, 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:02.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:02.467 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:18:02.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f9c0b66f0 msgr2=0x7f4f9c0b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f9c0b66f0 0x7f4f9c0b8c00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0004af0 tx=0x7f4fb0005d80 comp rx=0 tx=0).stop 2024-09-21T01:18:02.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 msgr2=0x7f4fb806d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb806d670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400e9f0 tx=0x7f4fa400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 shutdown_connections 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4f9c0b66f0 0x7f4f9c0b8c00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb8107a20 0x7f4fb8072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4fb8102f30 0x7f4fb806d670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 --2- 172.21.15.39:0/2311807073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fb81025e0 0x7f4fb806d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 >> 172.21.15.39:0/2311807073 conn(0x7f4fb80fe330 msgr2=0x7f4fb8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 shutdown_connections 2024-09-21T01:18:02.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:02.464+0000 7f4fb7e00640 1 -- 172.21.15.39:0/2311807073 wait complete. 2024-09-21T01:18:02.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 26.21 MB/sec warmup 29 sec latency 814.483 ms 2024-09-21T01:18:03.684 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b8d5ab3f', quiesce took 6.4 seconds. Will release after: 12.450000000000001 2024-09-21T01:18:03.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 25.34 MB/sec warmup 30 sec latency 1814.549 ms 2024-09-21T01:18:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:03 smithi150 ceph-mon[34945]: pgmap v841: 129 pgs: 129 active+clean; 751 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.4 MiB/s wr, 355 op/s 2024-09-21T01:18:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:03 smithi160 ceph-mon[34696]: pgmap v841: 129 pgs: 129 active+clean; 751 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.4 MiB/s wr, 355 op/s 2024-09-21T01:18:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:03 smithi039 ceph-mon[30415]: pgmap v841: 129 pgs: 129 active+clean; 751 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.4 MiB/s wr, 355 op/s 2024-09-21T01:18:04.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 24.52 MB/sec warmup 31 sec latency 2814.626 ms 2024-09-21T01:18:05.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 23.76 MB/sec warmup 32 sec latency 3814.759 ms 2024-09-21T01:18:05.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:05 smithi150 ceph-mon[34945]: pgmap v842: 129 pgs: 129 active+clean; 759 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 386 op/s 2024-09-21T01:18:05.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:05 smithi160 ceph-mon[34696]: pgmap v842: 129 pgs: 129 active+clean; 759 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 386 op/s 2024-09-21T01:18:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:05 smithi039 ceph-mon[30415]: pgmap v842: 129 pgs: 129 active+clean; 759 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 386 op/s 2024-09-21T01:18:06.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 23.04 MB/sec warmup 33 sec latency 4814.921 ms 2024-09-21T01:18:06.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:06 smithi150 ceph-mon[34945]: osdmap e107: 12 total, 12 up, 12 in 2024-09-21T01:18:06.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:06 smithi160 ceph-mon[34696]: osdmap e107: 12 total, 12 up, 12 in 2024-09-21T01:18:06.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:06 smithi039 ceph-mon[30415]: osdmap e107: 12 total, 12 up, 12 in 2024-09-21T01:18:07.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 22.36 MB/sec warmup 34 sec latency 5815.084 ms 2024-09-21T01:18:07.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:07 smithi150 ceph-mon[34945]: pgmap v844: 129 pgs: 129 active+clean; 762 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 376 op/s 2024-09-21T01:18:07.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:07 smithi160 ceph-mon[34696]: pgmap v844: 129 pgs: 129 active+clean; 762 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 376 op/s 2024-09-21T01:18:07.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:07 smithi039 ceph-mon[30415]: pgmap v844: 129 pgs: 129 active+clean; 762 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 376 op/s 2024-09-21T01:18:08.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 21.72 MB/sec warmup 35 sec latency 6815.199 ms 2024-09-21T01:18:09.740 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 21.12 MB/sec warmup 36 sec latency 7815.261 ms 2024-09-21T01:18:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:09 smithi150 ceph-mon[34945]: pgmap v845: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.5 MiB/s wr, 363 op/s 2024-09-21T01:18:09.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:09 smithi160 ceph-mon[34696]: pgmap v845: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.5 MiB/s wr, 363 op/s 2024-09-21T01:18:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:09 smithi039 ceph-mon[30415]: pgmap v845: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.5 MiB/s wr, 363 op/s 2024-09-21T01:18:10.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 20.55 MB/sec warmup 37 sec latency 8815.409 ms 2024-09-21T01:18:11.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 20.01 MB/sec warmup 38 sec latency 9815.577 ms 2024-09-21T01:18:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:11 smithi150 ceph-mon[34945]: pgmap v846: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 265 op/s 2024-09-21T01:18:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:11 smithi160 ceph-mon[34696]: pgmap v846: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 265 op/s 2024-09-21T01:18:11.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:11 smithi039 ceph-mon[30415]: pgmap v846: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 265 op/s 2024-09-21T01:18:12.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 19.50 MB/sec warmup 39 sec latency 10815.648 ms 2024-09-21T01:18:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:18:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:18:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:18:12.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:18:12.856 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:18:12.856 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:18:12.856 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:18:12.856 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:18:12.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:18:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:18:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:18:12.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:18:13.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 19.01 MB/sec warmup 40 sec latency 11815.694 ms 2024-09-21T01:18:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:13 smithi150 ceph-mon[34945]: pgmap v847: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2024-09-21T01:18:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:13 smithi160 ceph-mon[34696]: pgmap v847: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2024-09-21T01:18:13.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:13 smithi039 ceph-mon[30415]: pgmap v847: 129 pgs: 129 active+clean; 767 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2024-09-21T01:18:14.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 18.55 MB/sec warmup 41 sec latency 12815.740 ms 2024-09-21T01:18:15.721 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:15 smithi039 ceph-mon[30415]: pgmap v848: 129 pgs: 129 active+clean; 772 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.7 MiB/s wr, 109 op/s 2024-09-21T01:18:15.721 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:15 smithi039 ceph-mon[30415]: osdmap e108: 12 total, 12 up, 12 in 2024-09-21T01:18:15.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 18.10 MB/sec warmup 42 sec latency 13815.793 ms 2024-09-21T01:18:15.813 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:15 smithi150 ceph-mon[34945]: pgmap v848: 129 pgs: 129 active+clean; 772 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.7 MiB/s wr, 109 op/s 2024-09-21T01:18:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:15 smithi150 ceph-mon[34945]: osdmap e108: 12 total, 12 up, 12 in 2024-09-21T01:18:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:15 smithi160 ceph-mon[34696]: pgmap v848: 129 pgs: 129 active+clean; 772 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.7 MiB/s wr, 109 op/s 2024-09-21T01:18:15.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:15 smithi160 ceph-mon[34696]: osdmap e108: 12 total, 12 up, 12 in 2024-09-21T01:18:16.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:18:16.516 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 17.68 MB/sec warmup 43 sec latency 14815.844 ms 2024-09-21T01:18:17.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- 172.21.15.39:0/3221356798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01092f0 msgr2=0x7f49a0109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:17.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- 172.21.15.39:0/3221356798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01092f0 0x7f49a0109700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4990009900 tx=0x7f499002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- 172.21.15.39:0/3221356798 shutdown_connections 2024-09-21T01:18:17.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- 172.21.15.39:0/3221356798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49a0109c40 0x7f49a0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- 172.21.15.39:0/3221356798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01092f0 0x7f49a0109700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- 172.21.15.39:0/3221356798 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f49a01039b0 0x7f49a0103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- 172.21.15.39:0/3221356798 >> 172.21.15.39:0/3221356798 conn(0x7f49a00ff440 msgr2=0x7f49a0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- 172.21.15.39:0/3221356798 shutdown_connections 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- 172.21.15.39:0/3221356798 wait complete. 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 Processor -- start 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- start start 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 0x7f49a019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:17.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f49a01092f0 0x7f49a019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49a0109c40 0x7f49a019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f49a0113900 con 0x7f49a0109c40 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f49a0113780 con 0x7f49a01039b0 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.427+0000 7f49a7200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f49a0113a80 con 0x7f49a01092f0 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 0x7f49a019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 0x7f49a019a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:53092/0 (socket says 172.21.15.39:53092) 2024-09-21T01:18:17.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 -- 172.21.15.39:0/505795286 learned_addr learned my addr 172.21.15.39:0/505795286 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a6800640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49a0109c40 0x7f49a019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5400640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f49a01092f0 0x7f49a019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 -- 172.21.15.39:0/505795286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f49a01092f0 msgr2=0x7f49a019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f49a01092f0 0x7f49a019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 -- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49a0109c40 msgr2=0x7f49a019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49a0109c40 0x7f49a019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 -- 172.21.15.39:0/505795286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a01a0300 con 0x7f49a01039b0 2024-09-21T01:18:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a5e00640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 0x7f49a019a390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f499400ea60 tx=0x7f499400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499400cde0 con 0x7f49a01039b0 2024-09-21T01:18:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49940035f0 con 0x7f49a01039b0 2024-09-21T01:18:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994010690 con 0x7f49a01039b0 2024-09-21T01:18:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a01a05f0 con 0x7f49a01039b0 2024-09-21T01:18:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a00755f0 con 0x7f49a01039b0 2024-09-21T01:18:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.432+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a0104c30 con 0x7f49a01039b0 2024-09-21T01:18:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.437+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f49940037a0 con 0x7f49a01039b0 2024-09-21T01:18:17.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.437+0000 7f499aa00640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f49880b6680 0x7f49880b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:17.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.437+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7f49940e8140 con 0x7f49a01039b0 2024-09-21T01:18:17.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.437+0000 7f49a5400640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f49880b6680 0x7f49880b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:17.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.437+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4994056710 con 0x7f49a01039b0 2024-09-21T01:18:17.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.437+0000 7f49a5400640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f49880b6680 0x7f49880b8b90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4990009230 tx=0x7f4990031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:17.693 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:17 smithi039 ceph-mon[30415]: pgmap v850: 129 pgs: 129 active+clean; 777 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:18:17.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.700+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49a0103dc0 con 0x7f49a01039b0 2024-09-21T01:18:17.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.701+0000 7f499aa00640 1 -- 172.21.15.39:0/505795286 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11918 (secure 0 0 0) 0x7f49940ae250 con 0x7f49a01039b0 2024-09-21T01:18:17.703 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:18:17.703 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":50,"btime":"2024-09-21T01:18:00:503444+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:17:59.532100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:18:17.703 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 50 2024-09-21T01:18:17.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f49880b6680 msgr2=0x7f49880b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:17.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f49880b6680 0x7f49880b8b90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4990009230 tx=0x7f4990031400 comp rx=0 tx=0).stop 2024-09-21T01:18:17.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 msgr2=0x7f49a019a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:17.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 0x7f49a019a390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f499400ea60 tx=0x7f499400ef20 comp rx=0 tx=0).stop 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 shutdown_connections 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49a0109c40 0x7f49a019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f49a01092f0 0x7f49a019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f49880b6680 0x7f49880b8b90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 --2- 172.21.15.39:0/505795286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a01039b0 0x7f49a019a390 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 >> 172.21.15.39:0/505795286 conn(0x7f49a00ff440 msgr2=0x7f49a0101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 shutdown_connections 2024-09-21T01:18:17.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:17.704+0000 7f49a7200640 1 -- 172.21.15.39:0/505795286 wait complete. 2024-09-21T01:18:17.742 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 17.28 MB/sec warmup 44 sec latency 15815.900 ms 2024-09-21T01:18:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:17 smithi150 ceph-mon[34945]: pgmap v850: 129 pgs: 129 active+clean; 777 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:18:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:17 smithi160 ceph-mon[34696]: pgmap v850: 129 pgs: 129 active+clean; 777 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:18:18.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 16.90 MB/sec warmup 45 sec latency 16819.598 ms 2024-09-21T01:18:18.780 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:18 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/505795286' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:18.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:18 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/505795286' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:18.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:18 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/505795286' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:18.934 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id b8d5ab3f --release --await' 2024-09-21T01:18:18.934 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id b8d5ab3f --release --await 2024-09-21T01:18:19.366 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:19.652 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:19 smithi039 ceph-mon[30415]: pgmap v851: 129 pgs: 129 active+clean; 787 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 4.4 MiB/s wr, 118 op/s 2024-09-21T01:18:19.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10925 16.53 MB/sec warmup 46 sec latency 17819.638 ms 2024-09-21T01:18:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:19 smithi150 ceph-mon[34945]: pgmap v851: 129 pgs: 129 active+clean; 787 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 4.4 MiB/s wr, 118 op/s 2024-09-21T01:18:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:19 smithi160 ceph-mon[34696]: pgmap v851: 129 pgs: 129 active+clean; 787 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 4.4 MiB/s wr, 118 op/s 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 -- 172.21.15.39:0/952560156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1481050c0 msgr2=0x7fb14810d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 --2- 172.21.15.39:0/952560156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1481050c0 0x7fb14810d700 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb134009900 tx=0x7fb13402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 -- 172.21.15.39:0/952560156 shutdown_connections 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 --2- 172.21.15.39:0/952560156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb14810dc40 0x7fb148110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 --2- 172.21.15.39:0/952560156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1481050c0 0x7fb14810d700 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 --2- 172.21.15.39:0/952560156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb148104690 0x7fb148104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 -- 172.21.15.39:0/952560156 >> 172.21.15.39:0/952560156 conn(0x7fb1480fe2c0 msgr2=0x7fb148100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 -- 172.21.15.39:0/952560156 shutdown_connections 2024-09-21T01:18:20.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 -- 172.21.15.39:0/952560156 wait complete. 2024-09-21T01:18:20.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.255+0000 7fb14d200640 1 Processor -- start 2024-09-21T01:18:20.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 -- start start 2024-09-21T01:18:20.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb148104690 0x7fb1481025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481050c0 0x7fb148102ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14810dc40 0x7fb148103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb148113890 con 0x7fb14810dc40 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb148113710 con 0x7fb1481050c0 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb14d200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb148113a10 con 0x7fb148104690 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb148104690 0x7fb1481025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb148104690 0x7fb1481025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42174/0 (socket says 172.21.15.39:42174) 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147400640 1 -- 172.21.15.39:0/1683805892 learned_addr learned my addr 172.21.15.39:0/1683805892 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:20.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14810dc40 0x7fb148103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb146a00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481050c0 0x7fb148102ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 -- 172.21.15.39:0/1683805892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb148104690 msgr2=0x7fb1481025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb148104690 0x7fb1481025a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 -- 172.21.15.39:0/1683805892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481050c0 msgr2=0x7fb148102ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481050c0 0x7fb148102ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 -- 172.21.15.39:0/1683805892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1481a8580 con 0x7fb14810dc40 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb147e00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14810dc40 0x7fb148103020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb14000ec50 tx=0x7fb14000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:20.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb13fe00640 1 -- 172.21.15.39:0/1683805892 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140003b70 con 0x7fb14810dc40 2024-09-21T01:18:20.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb13fe00640 1 -- 172.21.15.39:0/1683805892 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb140003d10 con 0x7fb14810dc40 2024-09-21T01:18:20.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.260+0000 7fb13fe00640 1 -- 172.21.15.39:0/1683805892 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140010830 con 0x7fb14810dc40 2024-09-21T01:18:20.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1481a8870 con 0x7fb14810dc40 2024-09-21T01:18:20.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1481a8d30 con 0x7fb14810dc40 2024-09-21T01:18:20.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 Processor -- start 2024-09-21T01:18:20.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb13fe00640 1 -- 172.21.15.39:0/1683805892 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fb140003ec0 con 0x7fb14810dc40 2024-09-21T01:18:20.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 -- start start 2024-09-21T01:18:20.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1ee520 0x557cbe595340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 0x557cbe595880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe595dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557cbe50c290 con 0x557cbe1ee520 2024-09-21T01:18:20.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557cbe4fb6c0 con 0x557cbe1cd840 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.263+0000 7fb155470740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557cbe4ff4d0 con 0x557cbe1c5660 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 0x557cbe595880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 0x557cbe595880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:42180/0 (socket says 172.21.15.39:42180) 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 -- 172.21.15.39:0/1810777661 learned_addr learned my addr 172.21.15.39:0/1810777661 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb146a00640 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe595dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb146a00640 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1ee520 0x557cbe595340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 -- 172.21.15.39:0/1810777661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 msgr2=0x557cbe595dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:20.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe595dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 -- 172.21.15.39:0/1810777661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1ee520 msgr2=0x557cbe595340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:20.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1ee520 0x557cbe595340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 -- 172.21.15.39:0/1810777661 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cbe488460 con 0x557cbe1c5660 2024-09-21T01:18:20.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb13fe00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb12c0b6700 0x7fb12c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb147400640 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 0x557cbe595880 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb138009cd0 tx=0x7fb1380117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccf286dcd9217599 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:20.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb13fe00640 1 -- 172.21.15.39:0/1683805892 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fb1400e8100 con 0x7fb14810dc40 2024-09-21T01:18:20.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb13cc00640 1 -- 172.21.15.39:0/1810777661 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138012450 con 0x557cbe1c5660 2024-09-21T01:18:20.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb13cc00640 1 -- 172.21.15.39:0/1810777661 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb138012aa0 con 0x557cbe1c5660 2024-09-21T01:18:20.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb146a00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb12c0b6700 0x7fb12c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb13cc00640 1 -- 172.21.15.39:0/1810777661 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13800fc00 con 0x557cbe1c5660 2024-09-21T01:18:20.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 -- 172.21.15.39:0/1810777661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 msgr2=0x557cbe595880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 0x557cbe595880 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb138009cd0 tx=0x7fb1380117b0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb146a00640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb12c0b6700 0x7fb12c0b8c10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb134031440 tx=0x7fb134005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:20.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 -- 172.21.15.39:0/1810777661 shutdown_connections 2024-09-21T01:18:20.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe595dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1c5660 0x557cbe595880 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 --2- 172.21.15.39:0/1810777661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1ee520 0x557cbe595340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 -- 172.21.15.39:0/1810777661 >> 172.21.15.39:0/1810777661 conn(0x557cbe1e2910 msgr2=0x557cbe593230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:20.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 -- 172.21.15.39:0/1810777661 shutdown_connections 2024-09-21T01:18:20.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 -- 172.21.15.39:0/1810777661 wait complete. 2024-09-21T01:18:20.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 -- start start 2024-09-21T01:18:20.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.268+0000 7fb155470740 1 Processor -- start 2024-09-21T01:18:20.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe5a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe5aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 0x557cbe5af410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557cbe35cbb0 con 0x557cbe1c5660 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557cbe4966c0 con 0x557cbe1cd840 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557cbe5a3b20 con 0x557cbe1ee520 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb155470740 10 client.0 _ll_register_callbacks cb 0x557cbe4b1490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb147400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe5a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb147400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe5a9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47926/0 (socket says 172.21.15.39:47926) 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.273+0000 7fb147400640 1 -- 172.21.15.39:0/1447911648 learned_addr learned my addr 172.21.15.39:0/1447911648 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:20.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb147400640 1 -- 172.21.15.39:0/1447911648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 msgr2=0x557cbe5af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb147400640 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 0x557cbe5af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb147400640 1 -- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 msgr2=0x557cbe5aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb147400640 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe5aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb147400640 1 -- 172.21.15.39:0/1447911648 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cbe259b00 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb147400640 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe5a9e10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb13800f780 tx=0x7fb13800f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feab72ba594870c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb131600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb131600640 1 -- 172.21.15.39:0/1447911648 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138028070 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb138012aa0 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb131600640 1 -- 172.21.15.39:0/1447911648 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fb138004240 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb131600640 1 -- 172.21.15.39:0/1447911648 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fb138032050 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 10 client.16116 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557cbe2d0960 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb131600640 1 -- 172.21.15.39:0/1447911648 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb138012aa0 con 0x557cbe1c5660 2024-09-21T01:18:20.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb131600640 1 -- 172.21.15.39:0/1447911648 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7fb1380035f0 con 0x557cbe1c5660 2024-09-21T01:18:20.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 10 client.16116 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:20.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 10 client.16116 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:20.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 20 client.16116 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:20.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe1acea0 0x557cbe5bb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 4 client.16116 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:18:20.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.274+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cbe50e1e0 con 0x557cbe1acea0 2024-09-21T01:18:20.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.298+0000 7fb146a00640 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe1acea0 0x557cbe5bb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb146a00640 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe1acea0 0x557cbe5bb5b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94a30e126757ecdb server_cookie=89e90a7f9dbea05c in_seq=0 out_seq=0 2024-09-21T01:18:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb131600640 10 client.16116 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:20.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb131600640 1 -- 172.21.15.39:0/1447911648 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557cbe50e1e0 con 0x557cbe1acea0 2024-09-21T01:18:20.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb131600640 10 client.16116 handle_command_reply: tid=0 2024-09-21T01:18:20.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 client.16116 shutdown 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 msgr2=0x557cbe5a9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe5a9e10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb13800f780 tx=0x7fb13800f5c0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 shutdown_connections 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe1acea0 0x557cbe5bb5b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 0x557cbe5af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1cd840 0x557cbe5aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 --2- 172.21.15.39:0/1447911648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe5a9e10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 >> 172.21.15.39:0/1447911648 conn(0x557cbe1e2910 msgr2=0x557cbe598520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb131600640 0 client.16116 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 shutdown_connections 2024-09-21T01:18:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 1 -- 172.21.15.39:0/1447911648 wait complete. 2024-09-21T01:18:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.333+0000 7fb155470740 20 client.16116 trim_cache size 0 max 16384 2024-09-21T01:18:20.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 Processor -- start 2024-09-21T01:18:20.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- start start 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe592520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1cd840 0x557cbe592b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1ee520 0x557cbe593090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b1580 con 0x557cbe1c5660 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481aedc0 con 0x557cbe1ee520 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481acc70 con 0x557cbe1cd840 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe592520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe592520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47930/0 (socket says 172.21.15.39:47930) 2024-09-21T01:18:20.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 -- 172.21.15.39:0/1591131456 learned_addr learned my addr 172.21.15.39:0/1591131456 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:20.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 -- 172.21.15.39:0/1591131456 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1cd840 msgr2=0x557cbe592b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:18:20.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1cd840 0x557cbe592b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 -- 172.21.15.39:0/1591131456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1ee520 msgr2=0x557cbe593090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:18:20.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1ee520 0x557cbe593090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 -- 172.21.15.39:0/1591131456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cbe5a5510 con 0x557cbe1c5660 2024-09-21T01:18:20.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb147400640 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe592520 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb138003bb0 tx=0x7fb138009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dec4a689063aae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:20.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb133e00640 1 -- 172.21.15.39:0/1591131456 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138028040 con 0x557cbe1c5660 2024-09-21T01:18:20.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb133e00640 1 -- 172.21.15.39:0/1591131456 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb138005590 con 0x557cbe1c5660 2024-09-21T01:18:20.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb133e00640 1 -- 172.21.15.39:0/1591131456 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13800fc20 con 0x557cbe1c5660 2024-09-21T01:18:20.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- 172.21.15.39:0/1591131456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 msgr2=0x557cbe592520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe592520 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb138003bb0 tx=0x7fb138009790 comp rx=0 tx=0).stop 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- 172.21.15.39:0/1591131456 shutdown_connections 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1ee520 0x557cbe593090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1cd840 0x557cbe592b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 --2- 172.21.15.39:0/1591131456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1c5660 0x557cbe592520 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- 172.21.15.39:0/1591131456 >> 172.21.15.39:0/1591131456 conn(0x557cbe1e2910 msgr2=0x557cbe5a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- 172.21.15.39:0/1591131456 shutdown_connections 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- 172.21.15.39:0/1591131456 wait complete. 2024-09-21T01:18:20.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 -- start start 2024-09-21T01:18:20.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.356+0000 7fb155470740 1 Processor -- start 2024-09-21T01:18:20.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 0x557cbe5b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1cd840 0x557cbe5b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 0x557cbe624db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ad2c0 con 0x557cbe1cd840 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ae6f0 con 0x557cbe1c5660 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481ad950 con 0x557cbe1ee520 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 10 client.0 _ll_register_callbacks cb 0x7fb138009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 0x557cbe5b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 0x557cbe5b1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:53134/0 (socket says 172.21.15.39:53134) 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 -- 172.21.15.39:0/2429775751 learned_addr learned my addr 172.21.15.39:0/2429775751 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:20.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb146a00640 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1cd840 0x557cbe5b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 -- 172.21.15.39:0/2429775751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 msgr2=0x557cbe624db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:18:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 0x557cbe624db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 -- 172.21.15.39:0/2429775751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1cd840 msgr2=0x557cbe5b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1cd840 0x557cbe5b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 -- 172.21.15.39:0/2429775751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cbe6252f0 con 0x557cbe1c5660 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb147400640 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 0x557cbe5b1c90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb13800f8c0 tx=0x7fb138003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1719af0bd20016d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb131600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb131600640 1 -- 172.21.15.39:0/2429775751 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138028040 con 0x557cbe1c5660 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb138005590 con 0x557cbe1c5660 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb131600640 1 -- 172.21.15.39:0/2429775751 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fb13800f960 con 0x557cbe1c5660 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb131600640 1 -- 172.21.15.39:0/2429775751 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fb13800fb80 con 0x557cbe1c5660 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 10 client.26150 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557cbe6264b0 con 0x557cbe1c5660 2024-09-21T01:18:20.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb131600640 1 -- 172.21.15.39:0/2429775751 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb138005590 con 0x557cbe1c5660 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb131600640 1 -- 172.21.15.39:0/2429775751 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7fb138012450 con 0x557cbe1c5660 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 10 client.26150 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 10 client.26150 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 20 client.26150 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe182250 0x557cbe628a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 4 client.26150 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b8d5ab3f", "release": true, "await": true}] 2024-09-21T01:18:20.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.361+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b8d5ab3f", "release": true, "await": true}) -- 0x557cbe509c90 con 0x557cbe182250 2024-09-21T01:18:20.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.380+0000 7fb146a00640 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe182250 0x557cbe628a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:20.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.416+0000 7fb146a00640 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe182250 0x557cbe628a20 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96715df2ffdae71a server_cookie=c5739a4fbb64a481 in_seq=0 out_seq=0 2024-09-21T01:18:20.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.416+0000 7fb131600640 10 client.26150 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:20.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb131600640 1 -- 172.21.15.39:0/2429775751 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557cbe509c90 con 0x557cbe182250 2024-09-21T01:18:20.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb131600640 10 client.26150 handle_command_reply: tid=0 2024-09-21T01:18:20.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 client.26150 shutdown 2024-09-21T01:18:20.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 msgr2=0x557cbe5b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 0x557cbe5b1c90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb13800f8c0 tx=0x7fb138003820 comp rx=0 tx=0).stop 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 shutdown_connections 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557cbe182250 0x557cbe628a20 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557cbe1ee520 0x557cbe624db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557cbe1cd840 0x557cbe5b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 --2- 172.21.15.39:0/2429775751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cbe1c5660 0x557cbe5b1c90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 >> 172.21.15.39:0/2429775751 conn(0x557cbe1e2910 msgr2=0x557cbe592b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb131600640 0 client.26150 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 shutdown_connections 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 1 -- 172.21.15.39:0/2429775751 wait complete. 2024-09-21T01:18:20.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb155470740 20 client.26150 trim_cache size 0 max 16384 2024-09-21T01:18:20.455 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:18:20.455 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 50, 2024-09-21T01:18:20.455 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:18:20.455 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 56, 2024-09-21T01:18:20.455 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:18:20.455 INFO:teuthology.orchestra.run.smithi039.stdout: "b8d5ab3f": { 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 56, 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 138.8, 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:18:20.456 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb12c0b6700 msgr2=0x7fb12c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb12c0b6700 0x7fb12c0b8c10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb134031440 tx=0x7fb134005a20 comp rx=0 tx=0).stop 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14810dc40 msgr2=0x7fb148103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14810dc40 0x7fb148103020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb14000ec50 tx=0x7fb14000c6a0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 shutdown_connections 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14810dc40 0x7fb148103020 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb12c0b6700 0x7fb12c0b8c10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481050c0 0x7fb148102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 --2- 172.21.15.39:0/1683805892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb148104690 0x7fb1481025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 >> 172.21.15.39:0/1683805892 conn(0x7fb1480fe2c0 msgr2=0x7fb1481077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 shutdown_connections 2024-09-21T01:18:20.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:20.451+0000 7fb14d200640 1 -- 172.21.15.39:0/1683805892 wait complete. 2024-09-21T01:18:20.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10943 16.22 MB/sec warmup 47 sec latency 18702.584 ms 2024-09-21T01:18:21.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 10943 15.89 MB/sec warmup 48 sec latency 19702.640 ms 2024-09-21T01:18:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:21 smithi160 ceph-mon[34696]: pgmap v852: 129 pgs: 129 active+clean; 789 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 5.3 MiB/s wr, 137 op/s 2024-09-21T01:18:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:21 smithi039 ceph-mon[30415]: pgmap v852: 129 pgs: 129 active+clean; 789 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 5.3 MiB/s wr, 137 op/s 2024-09-21T01:18:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:21 smithi150 ceph-mon[34945]: pgmap v852: 129 pgs: 129 active+clean; 789 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 5.3 MiB/s wr, 137 op/s 2024-09-21T01:18:22.732 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b8d5ab3f', total seconds elapsed: 25.5 2024-09-21T01:18:22.732 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.85 2024-09-21T01:18:22.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:18:22.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 11106 15.67 MB/sec warmup 49 sec latency 20702.681 ms 2024-09-21T01:18:23.234 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:23.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 11594 15.68 MB/sec warmup 50 sec latency 21246.489 ms 2024-09-21T01:18:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:23 smithi160 ceph-mon[34696]: pgmap v853: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 6.4 MiB/s wr, 166 op/s 2024-09-21T01:18:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:23 smithi039 ceph-mon[30415]: pgmap v853: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 6.4 MiB/s wr, 166 op/s 2024-09-21T01:18:24.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:23 smithi150 ceph-mon[34945]: pgmap v853: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 6.4 MiB/s wr, 166 op/s 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.105+0000 7fc3fb600640 1 -- 172.21.15.39:0/2890927017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 msgr2=0x7fc3f410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.105+0000 7fc3fb600640 1 --2- 172.21.15.39:0/2890927017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f410f520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009be0 tx=0x7fc3ec02a6f0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.106+0000 7fc3fb600640 1 -- 172.21.15.39:0/2890927017 shutdown_connections 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.106+0000 7fc3fb600640 1 --2- 172.21.15.39:0/2890927017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f410f520 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.106+0000 7fc3fb600640 1 --2- 172.21.15.39:0/2890927017 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f41083a0 0x7fc3f41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.106+0000 7fc3fb600640 1 --2- 172.21.15.39:0/2890927017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f4102a60 0x7fc3f4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.106+0000 7fc3fb600640 1 -- 172.21.15.39:0/2890927017 >> 172.21.15.39:0/2890927017 conn(0x7fc3f4079730 msgr2=0x7fc3f4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:24.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.107+0000 7fc3fb600640 1 -- 172.21.15.39:0/2890927017 shutdown_connections 2024-09-21T01:18:24.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.108+0000 7fc3fb600640 1 -- 172.21.15.39:0/2890927017 wait complete. 2024-09-21T01:18:24.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.109+0000 7fc3fb600640 1 Processor -- start 2024-09-21T01:18:24.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.110+0000 7fc3fb600640 1 -- start start 2024-09-21T01:18:24.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.110+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f4102a60 0x7fc3f4111ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:24.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.111+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f41083a0 0x7fc3f4111ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:24.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.111+0000 7fc3f8c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f4102a60 0x7fc3f4111ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:24.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.111+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f4112530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:24.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.111+0000 7fc3f8c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f4102a60 0x7fc3f4111ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43970/0 (socket says 172.21.15.39:43970) 2024-09-21T01:18:24.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.111+0000 7fc3f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f41083a0 0x7fc3f4111ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:24.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.111+0000 7fc3f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f41083a0 0x7fc3f4111ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53192/0 (socket says 172.21.15.39:53192) 2024-09-21T01:18:24.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3fb600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f406bd60 con 0x7fc3f41083a0 2024-09-21T01:18:24.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3fb600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f406bbe0 con 0x7fc3f4108cf0 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3fb600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4113cc0 con 0x7fc3f4102a60 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f8c00640 1 -- 172.21.15.39:0/118857709 learned_addr learned my addr 172.21.15.39:0/118857709 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f9600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f4112530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f9600640 1 -- 172.21.15.39:0/118857709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f4102a60 msgr2=0x7fc3f4111ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f9600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f4102a60 0x7fc3f4111ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f9600640 1 -- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f41083a0 msgr2=0x7fc3f4111ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f9600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f41083a0 0x7fc3f4111ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.112+0000 7fc3f9600640 1 -- 172.21.15.39:0/118857709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f4074110 con 0x7fc3f4108cf0 2024-09-21T01:18:24.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.113+0000 7fc3f9600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f4112530 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec02ace0 tx=0x7fc3ec0315d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:24.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.113+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec041020 con 0x7fc3f4108cf0 2024-09-21T01:18:24.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.114+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f40743a0 con 0x7fc3f4108cf0 2024-09-21T01:18:24.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.114+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ec030790 con 0x7fc3f4108cf0 2024-09-21T01:18:24.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.114+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec036040 con 0x7fc3f4108cf0 2024-09-21T01:18:24.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.114+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f40748d0 con 0x7fc3f4108cf0 2024-09-21T01:18:24.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.117+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f4102e70 con 0x7fc3f4108cf0 2024-09-21T01:18:24.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.121+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fc3ec044020 con 0x7fc3f4108cf0 2024-09-21T01:18:24.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.123+0000 7fc3f1600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3dc0b67b0 0x7fc3dc0b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:24.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.123+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fc3ec1085a0 con 0x7fc3f4108cf0 2024-09-21T01:18:24.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.125+0000 7fc3f8c00640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3dc0b67b0 0x7fc3dc0b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:24.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.128+0000 7fc3f8c00640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3dc0b67b0 0x7fc3dc0b8cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8005fd0 tx=0x7fc3e800b480 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:24.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.129+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc3ec04b030 con 0x7fc3f4108cf0 2024-09-21T01:18:24.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.568+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3f4075400 con 0x7fc3f4108cf0 2024-09-21T01:18:24.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.569+0000 7fc3f1600640 1 -- 172.21.15.39:0/118857709 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11918 (secure 0 0 0) 0x7fc3ec0ce6b0 con 0x7fc3f4108cf0 2024-09-21T01:18:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:18:24.574 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":50,"btime":"2024-09-21T01:18:00:503444+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:17:59.532100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:18:24.574 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 50 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.572+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3dc0b67b0 msgr2=0x7fc3dc0b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.572+0000 7fc3fb600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3dc0b67b0 0x7fc3dc0b8cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8005fd0 tx=0x7fc3e800b480 comp rx=0 tx=0).stop 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.572+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 msgr2=0x7fc3f4112530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.572+0000 7fc3fb600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f4112530 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec02ace0 tx=0x7fc3ec0315d0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 shutdown_connections 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fc3dc0b67b0 0x7fc3dc0b8cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f4108cf0 0x7fc3f4112530 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3f41083a0 0x7fc3f4111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 --2- 172.21.15.39:0/118857709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc3f4102a60 0x7fc3f4111ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 >> 172.21.15.39:0/118857709 conn(0x7fc3f4079730 msgr2=0x7fc3f41042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 shutdown_connections 2024-09-21T01:18:24.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:24.573+0000 7fc3fb600640 1 -- 172.21.15.39:0/118857709 wait complete. 2024-09-21T01:18:24.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 12479 16.61 MB/sec warmup 51 sec latency 83.448 ms 2024-09-21T01:18:24.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:24 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/118857709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:24.884 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:24 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/118857709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:24 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/118857709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:25.466 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0' 2024-09-21T01:18:25.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 138.85 --await-for 45.0 2024-09-21T01:18:25.711 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:25 smithi039 ceph-mon[30415]: pgmap v854: 129 pgs: 129 active+clean; 798 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 6.9 MiB/s wr, 212 op/s 2024-09-21T01:18:25.745 INFO:tasks.workunit.client.0.smithi039.stdout: 10 13428 17.29 MB/sec warmup 52 sec latency 69.416 ms 2024-09-21T01:18:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:25 smithi160 ceph-mon[34696]: pgmap v854: 129 pgs: 129 active+clean; 798 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 6.9 MiB/s wr, 212 op/s 2024-09-21T01:18:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:25 smithi150 ceph-mon[34945]: pgmap v854: 129 pgs: 129 active+clean; 798 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 6.9 MiB/s wr, 212 op/s 2024-09-21T01:18:26.131 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:26.753 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14199 17.75 MB/sec warmup 53 sec latency 201.906 ms 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.952+0000 7fd001e00640 1 -- 172.21.15.39:0/3214487295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 msgr2=0x7fcffc10b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.952+0000 7fd001e00640 1 --2- 172.21.15.39:0/3214487295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc10b910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009900 tx=0x7fcfe802a420 comp rx=0 tx=0).stop 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 -- 172.21.15.39:0/3214487295 shutdown_connections 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 --2- 172.21.15.39:0/3214487295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc10be50 0x7fcffc112680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 --2- 172.21.15.39:0/3214487295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc10b910 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 --2- 172.21.15.39:0/3214487295 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc105bc0 0x7fcffc105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 -- 172.21.15.39:0/3214487295 >> 172.21.15.39:0/3214487295 conn(0x7fcffc101610 msgr2=0x7fcffc103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 -- 172.21.15.39:0/3214487295 shutdown_connections 2024-09-21T01:18:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.953+0000 7fd001e00640 1 -- 172.21.15.39:0/3214487295 wait complete. 2024-09-21T01:18:26.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.954+0000 7fd001e00640 1 Processor -- start 2024-09-21T01:18:26.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.955+0000 7fd001e00640 1 -- start start 2024-09-21T01:18:26.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.955+0000 7fd001e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc105bc0 0x7fcffc111170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.955+0000 7fd001e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc1116b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.955+0000 7fd001e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 0x7fcffc1ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc1116b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc0784b0 con 0x7fcffc105bc0 2024-09-21T01:18:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc078330 con 0x7fcffc10b500 2024-09-21T01:18:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc078630 con 0x7fcffc10be50 2024-09-21T01:18:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd000a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc105bc0 0x7fcffc111170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 0x7fcffc1ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 0x7fcffc1ac500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44000/0 (socket says 172.21.15.39:44000) 2024-09-21T01:18:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001400640 1 -- 172.21.15.39:0/2593933762 learned_addr learned my addr 172.21.15.39:0/2593933762 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.956+0000 7fd001400640 1 -- 172.21.15.39:0/2593933762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 msgr2=0x7fcffc1116b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fd001400640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc1116b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fd001400640 1 -- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc105bc0 msgr2=0x7fcffc111170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fd001400640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc105bc0 0x7fcffc111170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fd001400640 1 -- 172.21.15.39:0/2593933762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcffc1acb10 con 0x7fcffc10be50 2024-09-21T01:18:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fd000a00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc105bc0 0x7fcffc111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:18:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fcffbe00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc1116b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:18:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.957+0000 7fd001400640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 0x7fcffc1ac500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b9a0 tx=0x7fcff400be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:26.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.958+0000 7fcff9600640 1 -- 172.21.15.39:0/2593933762 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4019070 con 0x7fcffc10be50 2024-09-21T01:18:26.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.958+0000 7fcff9600640 1 -- 172.21.15.39:0/2593933762 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff400cba0 con 0x7fcffc10be50 2024-09-21T01:18:26.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.958+0000 7fcff9600640 1 -- 172.21.15.39:0/2593933762 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40075d0 con 0x7fcffc10be50 2024-09-21T01:18:26.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.958+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcffc1ace00 con 0x7fcffc10be50 2024-09-21T01:18:26.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.958+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcffc1ad330 con 0x7fcffc10be50 2024-09-21T01:18:26.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.966+0000 7fd00a0ca740 1 Processor -- start 2024-09-21T01:18:26.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.968+0000 7fcff9600640 1 -- 172.21.15.39:0/2593933762 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fcff401c070 con 0x7fcffc10be50 2024-09-21T01:18:26.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.968+0000 7fd00a0ca740 1 -- start start 2024-09-21T01:18:26.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.969+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c59fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.969+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.969+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5a0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.969+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56391c515a90 con 0x56391c1f8850 2024-09-21T01:18:26.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.969+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56391c504ec0 con 0x56391c1d7b70 2024-09-21T01:18:26.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.969+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56391c508cd0 con 0x56391c1cf990 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c59fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c59fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53224/0 (socket says 172.21.15.39:53224) 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fcffbe00640 1 -- 172.21.15.39:0/866130156 learned_addr learned my addr 172.21.15.39:0/866130156 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fd000a00640 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fcffbe00640 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fd000a00640 1 -- 172.21.15.39:0/866130156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 msgr2=0x56391c5a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:26.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fd000a00640 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fd000a00640 1 -- 172.21.15.39:0/866130156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 msgr2=0x56391c59fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:26.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.970+0000 7fd000a00640 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c59fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.971+0000 7fd000a00640 1 -- 172.21.15.39:0/866130156 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56391c4aa9d0 con 0x56391c1cf990 2024-09-21T01:18:26.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.972+0000 7fd000a00640 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5a00d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcffc1129f0 tx=0x7fcff000bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67861a678c484ada server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:26.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.972+0000 7fcfedc00640 1 -- 172.21.15.39:0/866130156 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff000d950 con 0x56391c1cf990 2024-09-21T01:18:26.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.972+0000 7fcfedc00640 1 -- 172.21.15.39:0/866130156 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff0019070 con 0x56391c1cf990 2024-09-21T01:18:26.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.973+0000 7fd00a0ca740 1 -- 172.21.15.39:0/866130156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 msgr2=0x56391c5a00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.974+0000 7fcff9600640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfe40b67c0 0x7fcfe40b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.974+0000 7fcfedc00640 1 -- 172.21.15.39:0/866130156 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0014e40 con 0x56391c1cf990 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.974+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5a00d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcffc1129f0 tx=0x7fcff000bc80 comp rx=0 tx=0).stop 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.974+0000 7fcff9600640 1 -- 172.21.15.39:0/2593933762 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fcff40e70d0 con 0x7fcffc10be50 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.974+0000 7fcffbe00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfe40b67c0 0x7fcfe40b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.975+0000 7fd00a0ca740 1 -- 172.21.15.39:0/866130156 shutdown_connections 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.975+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.975+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5a00d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.975+0000 7fcffbe00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfe40b67c0 0x7fcfe40b8cd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80096c0 tx=0x7fcfe8031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:26.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.975+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/866130156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c59fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.976+0000 7fd00a0ca740 1 -- 172.21.15.39:0/866130156 >> 172.21.15.39:0/866130156 conn(0x56391c1ecc40 msgr2=0x56391c59da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.977+0000 7fd00a0ca740 1 -- 172.21.15.39:0/866130156 shutdown_connections 2024-09-21T01:18:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.977+0000 7fd00a0ca740 1 -- 172.21.15.39:0/866130156 wait complete. 2024-09-21T01:18:26.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.978+0000 7fd00a0ca740 1 -- start start 2024-09-21T01:18:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.978+0000 7fd00a0ca740 1 Processor -- start 2024-09-21T01:18:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c5ab340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 0x56391c5ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1f8850 0x56391c5b7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56391c3667b0 con 0x56391c1cf990 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x56391c4a9650 con 0x56391c1f8850 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56391c5ae3c0 con 0x56391c1d7b70 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fd00a0ca740 10 client.0 _ll_register_callbacks cb 0x56391c2100a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.979+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 0x56391c5ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.980+0000 7fd000a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c5ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.980+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 0x56391c5ab880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44006/0 (socket says 172.21.15.39:44006) 2024-09-21T01:18:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.980+0000 7fcffbe00640 1 -- 172.21.15.39:0/3698225618 learned_addr learned my addr 172.21.15.39:0/3698225618 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.981+0000 7fcffbe00640 1 -- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1f8850 msgr2=0x56391c5b7700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.981+0000 7fcffbe00640 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1f8850 0x56391c5b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.981+0000 7fcffbe00640 1 -- 172.21.15.39:0/3698225618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 msgr2=0x56391c5ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.981+0000 7fcffbe00640 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c5ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.981+0000 7fcffbe00640 1 -- 172.21.15.39:0/3698225618 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391c272ba0 con 0x56391c1d7b70 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.981+0000 7fcffbe00640 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 0x56391c5ab880 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8005dc0 tx=0x7fcfe8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196d037ea24d5b72 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.982+0000 7fcfe2000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.982+0000 7fcfe2000640 1 -- 172.21.15.39:0/3698225618 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe804e070 con 0x56391c1d7b70 2024-09-21T01:18:26.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.982+0000 7fcfe2000640 1 -- 172.21.15.39:0/3698225618 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fcfe8003ac0 con 0x56391c1d7b70 2024-09-21T01:18:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.982+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcff0019070 con 0x56391c1d7b70 2024-09-21T01:18:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.983+0000 7fcfe2000640 1 -- 172.21.15.39:0/3698225618 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fcfe8058050 con 0x56391c1d7b70 2024-09-21T01:18:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.983+0000 7fd00a0ca740 10 client.26152 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.983+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56391c5cb220 con 0x56391c1d7b70 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.983+0000 7fcfe2000640 1 -- 172.21.15.39:0/3698225618 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff0019070 con 0x56391c1d7b70 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.983+0000 7fcfe2000640 1 -- 172.21.15.39:0/3698225618 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7fcfe8053440 con 0x56391c1d7b70 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd00a0ca740 10 client.26152 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd00a0ca740 10 client.26152 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd00a0ca740 20 client.26152 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c1b71d0 0x56391c5ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd00a0ca740 4 client.26152 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56391c5179e0 con 0x56391c1b71d0 2024-09-21T01:18:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.984+0000 7fd000a00640 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c1b71d0 0x56391c5ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.985+0000 7fd000a00640 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c1b71d0 0x56391c5ce860 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a069c7cd7fd49ba3 server_cookie=cd6561b54051407e in_seq=0 out_seq=0 2024-09-21T01:18:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.985+0000 7fcfe2000640 10 client.26152 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.988+0000 7fcfe2000640 1 -- 172.21.15.39:0/3698225618 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56391c5179e0 con 0x56391c1b71d0 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.988+0000 7fcfe2000640 10 client.26152 handle_command_reply: tid=0 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.988+0000 7fd00a0ca740 1 client.26152 shutdown 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.988+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 msgr2=0x56391c5ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.988+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 0x56391c5ab880 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8005dc0 tx=0x7fcfe8005fd0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 shutdown_connections 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c1b71d0 0x56391c5ce860 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1f8850 0x56391c5b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1d7b70 0x56391c5ab880 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/3698225618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c5ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:26.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fcfe2000640 0 client.26152 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:26.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.989+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 >> 172.21.15.39:0/3698225618 conn(0x56391c1ecc40 msgr2=0x56391c5a2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:26.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.991+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 shutdown_connections 2024-09-21T01:18:26.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.991+0000 7fd00a0ca740 1 -- 172.21.15.39:0/3698225618 wait complete. 2024-09-21T01:18:26.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:26.991+0000 7fd00a0ca740 20 client.26152 trim_cache size 0 max 16384 2024-09-21T01:18:27.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.027+0000 7fd00a0ca740 1 Processor -- start 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.027+0000 7fd00a0ca740 1 -- start start 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.027+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.028+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.028+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c5a8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.028+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1b1650 con 0x56391c1f8850 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.028+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1aee90 con 0x56391c1d7b70 2024-09-21T01:18:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.028+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc106090 con 0x56391c1cf990 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.028+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.029+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:60376/0 (socket says 172.21.15.39:60376) 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.029+0000 7fd000a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.029+0000 7fd000a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5b7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44022/0 (socket says 172.21.15.39:44022) 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.029+0000 7fd000a00640 1 -- 172.21.15.39:0/798863718 learned_addr learned my addr 172.21.15.39:0/798863718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.029+0000 7fd001400640 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c5a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:27.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.029+0000 7fd000a00640 1 -- 172.21.15.39:0/798863718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 msgr2=0x56391c5ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fd000a00640 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fd000a00640 1 -- 172.21.15.39:0/798863718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 msgr2=0x56391c5a8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fd000a00640 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c5a8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fd000a00640 1 -- 172.21.15.39:0/798863718 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56391c5ac600 con 0x56391c1cf990 2024-09-21T01:18:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fd001400640 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c5a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:18:27.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fcffbe00640 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:18:27.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.030+0000 7fd000a00640 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5b7600 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcff00149f0 tx=0x7fcff0014980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62d53d83301f7e7d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.031+0000 7fcfed200640 1 -- 172.21.15.39:0/798863718 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0014a90 con 0x56391c1cf990 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.031+0000 7fcfed200640 1 -- 172.21.15.39:0/798863718 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff0019070 con 0x56391c1cf990 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.031+0000 7fcfed200640 1 -- 172.21.15.39:0/798863718 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0004460 con 0x56391c1cf990 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 -- 172.21.15.39:0/798863718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 msgr2=0x56391c5b7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5b7600 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcff00149f0 tx=0x7fcff0014980 comp rx=0 tx=0).stop 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 -- 172.21.15.39:0/798863718 shutdown_connections 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1f8850 0x56391c5a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c5ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/798863718 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1cf990 0x56391c5b7600 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 -- 172.21.15.39:0/798863718 >> 172.21.15.39:0/798863718 conn(0x56391c1ecc40 msgr2=0x56391c59c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.032+0000 7fd00a0ca740 1 -- 172.21.15.39:0/798863718 shutdown_connections 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.033+0000 7fd00a0ca740 1 -- 172.21.15.39:0/798863718 wait complete. 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.033+0000 7fd00a0ca740 1 -- start start 2024-09-21T01:18:27.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.033+0000 7fd00a0ca740 1 Processor -- start 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c632a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c632f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 0x56391c5b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc11d4c0 con 0x56391c1cf990 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1ae7c0 con 0x56391c1d7b70 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1ada20 con 0x56391c1f8850 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.034+0000 7fd00a0ca740 10 client.0 _ll_register_callbacks cb 0x7fcfe8005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:27.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 0x56391c5b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 0x56391c5b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44026/0 (socket says 172.21.15.39:44026) 2024-09-21T01:18:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 -- 172.21.15.39:0/1112614552 learned_addr learned my addr 172.21.15.39:0/1112614552 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:27.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd000a00640 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c632a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 -- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 msgr2=0x56391c632f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c632f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 -- 172.21.15.39:0/1112614552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 msgr2=0x56391c632a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c632a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.035+0000 7fd001400640 1 -- 172.21.15.39:0/1112614552 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391c5b3bf0 con 0x56391c1f8850 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.036+0000 7fd001400640 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 0x56391c5b3660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcff4013000 tx=0x7fcff40ac2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39087311cf890aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.036+0000 7fcfe2000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.036+0000 7fcfe2000640 1 -- 172.21.15.39:0/1112614552 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40e4d90 con 0x56391c1f8850 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.036+0000 7fcfe2000640 1 -- 172.21.15.39:0/1112614552 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fcff40ceb50 con 0x56391c1f8850 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.037+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcff0019070 con 0x56391c1f8850 2024-09-21T01:18:27.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.037+0000 7fcfe2000640 1 -- 172.21.15.39:0/1112614552 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fcff40ad240 con 0x56391c1f8850 2024-09-21T01:18:27.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.037+0000 7fd00a0ca740 10 client.26158 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:27.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.037+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56391c64aaf0 con 0x56391c1f8850 2024-09-21T01:18:27.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.037+0000 7fcfe2000640 1 -- 172.21.15.39:0/1112614552 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff0019070 con 0x56391c1f8850 2024-09-21T01:18:27.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.038+0000 7fcfe2000640 1 -- 172.21.15.39:0/1112614552 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7fcff40abda0 con 0x56391c1f8850 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.038+0000 7fd00a0ca740 10 client.26158 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.038+0000 7fd00a0ca740 10 client.26158 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.039+0000 7fd00a0ca740 20 client.26158 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.039+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c18c580 0x56391c64d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.039+0000 7fd00a0ca740 4 client.26158 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}] 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.039+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.85, "await_for": 45.0}) -- 0x56391c513490 con 0x56391c18c580 2024-09-21T01:18:27.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.039+0000 7fd000a00640 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c18c580 0x56391c64d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:27.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.040+0000 7fd000a00640 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c18c580 0x56391c64d330 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e133189b3e6026f0 server_cookie=2fc194435cd41a4c in_seq=0 out_seq=0 2024-09-21T01:18:27.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.040+0000 7fcfe2000640 10 client.26158 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fcfe2000640 1 -- 172.21.15.39:0/1112614552 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56391c513490 con 0x56391c18c580 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fcfe2000640 10 client.26158 handle_command_reply: tid=0 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 client.26158 shutdown 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 msgr2=0x56391c5b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 0x56391c5b3660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcff4013000 tx=0x7fcff40ac2b0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 shutdown_connections 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x56391c18c580 0x56391c64d330 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x56391c1f8850 0x56391c5b3660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56391c1d7b70 0x56391c632f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 --2- 172.21.15.39:0/1112614552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56391c1cf990 0x56391c632a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fcfe2000640 0 client.26158 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 >> 172.21.15.39:0/1112614552 conn(0x56391c1ecc40 msgr2=0x56391c5ac7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 shutdown_connections 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.661+0000 7fd00a0ca740 1 -- 172.21.15.39:0/1112614552 wait complete. 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd00a0ca740 20 client.26158 trim_cache size 0 max 16384 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 50, 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 58, 2024-09-21T01:18:27.664 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "6b596caf": { 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 58, 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 138.8, 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:27.665 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:27.666 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfe40b67c0 msgr2=0x7fcfe40b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfe40b67c0 0x7fcfe40b8cd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80096c0 tx=0x7fcfe8031400 comp rx=0 tx=0).stop 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 msgr2=0x7fcffc1ac500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 0x7fcffc1ac500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b9a0 tx=0x7fcff400be60 comp rx=0 tx=0).stop 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 shutdown_connections 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcffc10be50 0x7fcffc1ac500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcfe40b67c0 0x7fcfe40b8cd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcffc10b500 0x7fcffc1116b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.662+0000 7fd001e00640 1 --2- 172.21.15.39:0/2593933762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcffc105bc0 0x7fcffc111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.663+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 >> 172.21.15.39:0/2593933762 conn(0x7fcffc101610 msgr2=0x7fcffc07ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.663+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 shutdown_connections 2024-09-21T01:18:27.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:27.663+0000 7fd001e00640 1 -- 172.21.15.39:0/2593933762 wait complete. 2024-09-21T01:18:27.753 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 17.59 MB/sec warmup 54 sec latency 689.570 ms 2024-09-21T01:18:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:27 smithi039 ceph-mon[30415]: pgmap v855: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 5.7 MiB/s wr, 194 op/s 2024-09-21T01:18:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:27 smithi150 ceph-mon[34945]: pgmap v855: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 5.7 MiB/s wr, 194 op/s 2024-09-21T01:18:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:27 smithi160 ceph-mon[34696]: pgmap v855: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 5.7 MiB/s wr, 194 op/s 2024-09-21T01:18:28.618 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6b596caf', quiesce took 5.9 seconds. Will release after: 12.950000000000001 2024-09-21T01:18:28.753 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 17.27 MB/sec warmup 55 sec latency 1689.687 ms 2024-09-21T01:18:29.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 16.96 MB/sec warmup 56 sec latency 2689.833 ms 2024-09-21T01:18:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:29 smithi039 ceph-mon[30415]: pgmap v856: 129 pgs: 129 active+clean; 810 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.2 MiB/s wr, 337 op/s 2024-09-21T01:18:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:29 smithi150 ceph-mon[34945]: pgmap v856: 129 pgs: 129 active+clean; 810 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.2 MiB/s wr, 337 op/s 2024-09-21T01:18:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:29 smithi160 ceph-mon[34696]: pgmap v856: 129 pgs: 129 active+clean; 810 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.2 MiB/s wr, 337 op/s 2024-09-21T01:18:30.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 16.66 MB/sec warmup 57 sec latency 3690.021 ms 2024-09-21T01:18:31.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 16.37 MB/sec warmup 58 sec latency 4690.119 ms 2024-09-21T01:18:31.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:31 smithi039 ceph-mon[30415]: pgmap v857: 129 pgs: 129 active+clean; 818 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.3 MiB/s wr, 325 op/s 2024-09-21T01:18:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:31 smithi150 ceph-mon[34945]: pgmap v857: 129 pgs: 129 active+clean; 818 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.3 MiB/s wr, 325 op/s 2024-09-21T01:18:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:31 smithi160 ceph-mon[34696]: pgmap v857: 129 pgs: 129 active+clean; 818 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.3 MiB/s wr, 325 op/s 2024-09-21T01:18:32.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 16.10 MB/sec warmup 59 sec latency 5690.267 ms 2024-09-21T01:18:33.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 15.83 MB/sec warmup 60 sec latency 6690.342 ms 2024-09-21T01:18:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:33 smithi039 ceph-mon[30415]: pgmap v858: 129 pgs: 129 active+clean; 820 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 324 op/s 2024-09-21T01:18:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:33 smithi150 ceph-mon[34945]: pgmap v858: 129 pgs: 129 active+clean; 820 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 324 op/s 2024-09-21T01:18:34.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:33 smithi160 ceph-mon[34696]: pgmap v858: 129 pgs: 129 active+clean; 820 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 324 op/s 2024-09-21T01:18:34.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 15.57 MB/sec warmup 61 sec latency 7690.405 ms 2024-09-21T01:18:35.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 15.32 MB/sec warmup 62 sec latency 8690.536 ms 2024-09-21T01:18:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:35 smithi039 ceph-mon[30415]: pgmap v859: 129 pgs: 129 active+clean; 819 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 302 op/s 2024-09-21T01:18:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:35 smithi150 ceph-mon[34945]: pgmap v859: 129 pgs: 129 active+clean; 819 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 302 op/s 2024-09-21T01:18:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:35 smithi160 ceph-mon[34696]: pgmap v859: 129 pgs: 129 active+clean; 819 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 302 op/s 2024-09-21T01:18:36.754 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 15.08 MB/sec warmup 63 sec latency 9690.690 ms 2024-09-21T01:18:37.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 14.84 MB/sec warmup 64 sec latency 10690.820 ms 2024-09-21T01:18:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:37 smithi039 ceph-mon[30415]: pgmap v860: 129 pgs: 129 active+clean; 819 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 257 op/s 2024-09-21T01:18:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:37 smithi150 ceph-mon[34945]: pgmap v860: 129 pgs: 129 active+clean; 819 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 257 op/s 2024-09-21T01:18:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:37 smithi160 ceph-mon[34696]: pgmap v860: 129 pgs: 129 active+clean; 819 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 257 op/s 2024-09-21T01:18:38.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 14.61 MB/sec warmup 65 sec latency 11690.913 ms 2024-09-21T01:18:39.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 14.39 MB/sec warmup 66 sec latency 12690.958 ms 2024-09-21T01:18:39.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:39 smithi039 ceph-mon[30415]: pgmap v861: 129 pgs: 129 active+clean; 820 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-09-21T01:18:40.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:39 smithi150 ceph-mon[34945]: pgmap v861: 129 pgs: 129 active+clean; 820 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-09-21T01:18:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:39 smithi160 ceph-mon[34696]: pgmap v861: 129 pgs: 129 active+clean; 820 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-09-21T01:18:40.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 14.18 MB/sec warmup 67 sec latency 13691.074 ms 2024-09-21T01:18:41.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:18:41.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 13.97 MB/sec warmup 68 sec latency 14691.159 ms 2024-09-21T01:18:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:41 smithi039 ceph-mon[30415]: pgmap v862: 129 pgs: 129 active+clean; 821 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 MiB/s wr, 98 op/s 2024-09-21T01:18:41.962 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:41 smithi150 ceph-mon[34945]: pgmap v862: 129 pgs: 129 active+clean; 821 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 MiB/s wr, 98 op/s 2024-09-21T01:18:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:41 smithi160 ceph-mon[34696]: pgmap v862: 129 pgs: 129 active+clean; 821 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 MiB/s wr, 98 op/s 2024-09-21T01:18:42.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 -- 172.21.15.39:0/1073782940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58103b60 msgr2=0x7f4d58103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:42.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 --2- 172.21.15.39:0/1073782940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58103b60 0x7f4d58103f70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c009900 tx=0x7f4d4c02a410 comp rx=0 tx=0).stop 2024-09-21T01:18:42.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 -- 172.21.15.39:0/1073782940 shutdown_connections 2024-09-21T01:18:42.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 --2- 172.21.15.39:0/1073782940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58109df0 0x7f4d58110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 --2- 172.21.15.39:0/1073782940 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d581094a0 0x7f4d581098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 --2- 172.21.15.39:0/1073782940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58103b60 0x7f4d58103f70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 -- 172.21.15.39:0/1073782940 >> 172.21.15.39:0/1073782940 conn(0x7f4d5807b850 msgr2=0x7f4d5807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 -- 172.21.15.39:0/1073782940 shutdown_connections 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.588+0000 7f4d5f400640 1 -- 172.21.15.39:0/1073782940 wait complete. 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.589+0000 7f4d5f400640 1 Processor -- start 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.589+0000 7f4d5f400640 1 -- start start 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.589+0000 7f4d5f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58103b60 0x7f4d58075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5f400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d581094a0 0x7f4d58075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 0x7f4d58076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5f400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4d58113ab0 con 0x7f4d58109df0 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5f400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4d58113930 con 0x7f4d58103b60 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5f400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4d58113c30 con 0x7f4d581094a0 2024-09-21T01:18:42.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 0x7f4d58076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 0x7f4d58076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35044/0 (socket says 172.21.15.39:35044) 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 -- 172.21.15.39:0/2269874717 learned_addr learned my addr 172.21.15.39:0/2269874717 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5ca00640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58103b60 0x7f4d58075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d57e00640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d581094a0 0x7f4d58075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 -- 172.21.15.39:0/2269874717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d581094a0 msgr2=0x7f4d58075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d581094a0 0x7f4d58075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 -- 172.21.15.39:0/2269874717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58103b60 msgr2=0x7f4d58075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58103b60 0x7f4d58075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5d400640 1 -- 172.21.15.39:0/2269874717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d581a87e0 con 0x7f4d58109df0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.590+0000 7f4d5ca00640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58103b60 0x7f4d58075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.591+0000 7f4d5d400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 0x7f4d58076190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009fd0 tx=0x7f4d5000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.591+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50017070 con 0x7f4d58109df0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.591+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d50003e20 con 0x7f4d58109df0 2024-09-21T01:18:42.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.591+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d581a8ad0 con 0x7f4d58109df0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.591+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50010460 con 0x7f4d58109df0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.591+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d581a8fc0 con 0x7f4d58109df0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.592+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d58103f70 con 0x7f4d58109df0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.595+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4d50003970 con 0x7f4d58109df0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.596+0000 7f4d55600640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4d400b6680 0x7f4d400b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.596+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7f4d500e77b0 con 0x7f4d58109df0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.596+0000 7f4d5ca00640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4d400b6680 0x7f4d400b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.597+0000 7f4d5ca00640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4d400b6680 0x7f4d400b8b90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c000c00 tx=0x7f4d4c031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:42.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.600+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4d500b1ca0 con 0x7f4d58109df0 2024-09-21T01:18:42.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 13.77 MB/sec warmup 69 sec latency 15691.237 ms 2024-09-21T01:18:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.843+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d5802e640 con 0x7f4d58109df0 2024-09-21T01:18:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.844+0000 7f4d55600640 1 -- 172.21.15.39:0/2269874717 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11918 (secure 0 0 0) 0x7f4d500b6be0 con 0x7f4d58109df0 2024-09-21T01:18:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:18:42.846 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":50,"btime":"2024-09-21T01:18:00:503444+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:17:59.532100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:18:42.846 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 50 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4d400b6680 msgr2=0x7f4d400b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4d400b6680 0x7f4d400b8b90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c000c00 tx=0x7f4d4c031400 comp rx=0 tx=0).stop 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 msgr2=0x7f4d58076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 0x7f4d58076190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009fd0 tx=0x7f4d5000b260 comp rx=0 tx=0).stop 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 shutdown_connections 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d58109df0 0x7f4d58076190 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d581094a0 0x7f4d58075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f4d400b6680 0x7f4d400b8b90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 --2- 172.21.15.39:0/2269874717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d58103b60 0x7f4d58075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 >> 172.21.15.39:0/2269874717 conn(0x7f4d5807b850 msgr2=0x7f4d581075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 shutdown_connections 2024-09-21T01:18:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:42.847+0000 7f4d5f400640 1 -- 172.21.15.39:0/2269874717 wait complete. 2024-09-21T01:18:43.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 13.57 MB/sec warmup 70 sec latency 16691.295 ms 2024-09-21T01:18:43.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:43 smithi039 ceph-mon[30415]: pgmap v863: 129 pgs: 129 active+clean; 826 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-09-21T01:18:43.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:43 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2269874717' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:43.893 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 6b596caf --release --await' 2024-09-21T01:18:43.894 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 6b596caf --release --await 2024-09-21T01:18:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:43 smithi150 ceph-mon[34945]: pgmap v863: 129 pgs: 129 active+clean; 826 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-09-21T01:18:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:43 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2269874717' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:43 smithi160 ceph-mon[34696]: pgmap v863: 129 pgs: 129 active+clean; 826 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-09-21T01:18:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:43 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2269874717' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:18:44.280 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:18:44.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 13.38 MB/sec warmup 71 sec latency 17691.353 ms 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.731+0000 7fa022400640 1 -- 172.21.15.39:0/3884829960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 msgr2=0x7fa01c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.731+0000 7fa022400640 1 --2- 172.21.15.39:0/3884829960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c1098b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa008009900 tx=0x7fa00802a410 comp rx=0 tx=0).stop 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 -- 172.21.15.39:0/3884829960 shutdown_connections 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 --2- 172.21.15.39:0/3884829960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa01c109df0 0x7fa01c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 --2- 172.21.15.39:0/3884829960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c1098b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 --2- 172.21.15.39:0/3884829960 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 0x7fa01c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 -- 172.21.15.39:0/3884829960 >> 172.21.15.39:0/3884829960 conn(0x7fa01c07b850 msgr2=0x7fa01c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 -- 172.21.15.39:0/3884829960 shutdown_connections 2024-09-21T01:18:45.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.733+0000 7fa022400640 1 -- 172.21.15.39:0/3884829960 wait complete. 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 Processor -- start 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 -- start start 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 0x7fa01c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa01c109df0 0x7fa01c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c113b00 con 0x7fa01c109df0 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c113980 con 0x7fa01c1094a0 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c113c80 con 0x7fa01c103b60 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 0x7fa01c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 0x7fa01c075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44008/0 (socket says 172.21.15.39:44008) 2024-09-21T01:18:45.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01b400640 1 -- 172.21.15.39:0/664604939 learned_addr learned my addr 172.21.15.39:0/664604939 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:45.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01be00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa01c109df0 0x7fa01c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 -- 172.21.15.39:0/664604939 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 msgr2=0x7fa01c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 0x7fa01c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 -- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa01c109df0 msgr2=0x7fa01c076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa01c109df0 0x7fa01c076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 -- 172.21.15.39:0/664604939 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c1a86f0 con 0x7fa01c1094a0 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa01aa00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c075c50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa008009230 tx=0x7fa008003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa013e00640 1 -- 172.21.15.39:0/664604939 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00803f070 con 0x7fa01c1094a0 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01c1a8980 con 0x7fa01c1094a0 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa013e00640 1 -- 172.21.15.39:0/664604939 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa008031b50 con 0x7fa01c1094a0 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa013e00640 1 -- 172.21.15.39:0/664604939 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008035030 con 0x7fa01c1094a0 2024-09-21T01:18:45.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01c1a8e30 con 0x7fa01c1094a0 2024-09-21T01:18:45.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.734+0000 7fa0291bf740 1 Processor -- start 2024-09-21T01:18:45.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 -- start start 2024-09-21T01:18:45.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 0x55a4947b6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943ee840 0x55a4947b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a49472d290 con 0x55a4943ee840 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a49471c6c0 con 0x55a4943e6660 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa0291bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a4947204d0 con 0x55a49440f520 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 0x55a4947b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 0x55a4947b6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:55492/0 (socket says 172.21.15.39:55492) 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 -- 172.21.15.39:0/2536828415 learned_addr learned my addr 172.21.15.39:0/2536828415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01be00640 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa013e00640 1 -- 172.21.15.39:0/664604939 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa008042070 con 0x7fa01c1094a0 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 -- 172.21.15.39:0/2536828415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 msgr2=0x55a4947b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 -- 172.21.15.39:0/2536828415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943ee840 msgr2=0x55a4947b6c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943ee840 0x55a4947b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.740+0000 7fa01b400640 1 -- 172.21.15.39:0/2536828415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4946a9460 con 0x55a4943e6660 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa01be00640 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa01b400640 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 0x55a4947b6700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d70 tx=0x7fa00c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d830bba76e32984 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa010c00640 1 -- 172.21.15.39:0/2536828415 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c01e070 con 0x55a4943e6660 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa010c00640 1 -- 172.21.15.39:0/2536828415 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa00c019ae0 con 0x55a4943e6660 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa010c00640 1 -- 172.21.15.39:0/2536828415 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c00eb40 con 0x55a4943e6660 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa013e00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0000b6680 0x7fa0000b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 -- 172.21.15.39:0/2536828415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 msgr2=0x55a4947b6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 0x55a4947b6700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d70 tx=0x7fa00c011810 comp rx=0 tx=0).stop 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 -- 172.21.15.39:0/2536828415 shutdown_connections 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943ee840 0x55a4947b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943e6660 0x55a4947b6700 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2536828415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 -- 172.21.15.39:0/2536828415 >> 172.21.15.39:0/2536828415 conn(0x55a494403910 msgr2=0x55a4947b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa013e00640 1 -- 172.21.15.39:0/664604939 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fa008107d50 con 0x7fa01c1094a0 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa01be00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0000b6680 0x7fa0000b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 -- 172.21.15.39:0/2536828415 shutdown_connections 2024-09-21T01:18:45.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 -- 172.21.15.39:0/2536828415 wait complete. 2024-09-21T01:18:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa01be00640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0000b6680 0x7fa0000b8b90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa014005fd0 tx=0x7fa014009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 -- start start 2024-09-21T01:18:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.742+0000 7fa0291bf740 1 Processor -- start 2024-09-21T01:18:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943ee840 0x55a4947cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a49440f520 0x55a4947d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a49457dbb0 con 0x55a4943e6660 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a4946b76c0 con 0x55a49440f520 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55a4947c49a0 con 0x55a4943ee840 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa0291bf740 10 client.0 _ll_register_callbacks cb 0x55a4946d2490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947cac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34802/0 (socket says 172.21.15.39:34802) 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 -- 172.21.15.39:0/2406404844 learned_addr learned my addr 172.21.15.39:0/2406404844 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01be00640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a49440f520 0x55a4947d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01aa00640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943ee840 0x55a4947cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 -- 172.21.15.39:0/2406404844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943ee840 msgr2=0x55a4947cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943ee840 0x55a4947cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 -- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a49440f520 msgr2=0x55a4947d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a49440f520 0x55a4947d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.745+0000 7fa01b400640 1 -- 172.21.15.39:0/2406404844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a49447ab00 con 0x55a4943e6660 2024-09-21T01:18:45.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa01b400640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947cac40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009a90 tx=0x7fa00c009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc3817511c2ea99 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:45.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:18:45.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 1 -- 172.21.15.39:0/2406404844 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c029020 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa00c019ae0 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 1 -- 172.21.15.39:0/2406404844 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fa00c004380 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 1 -- 172.21.15.39:0/2406404844 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fa00c034050 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 10 client.16155 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55a4944f1960 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 1 -- 172.21.15.39:0/2406404844 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa00c019ae0 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 1 -- 172.21.15.39:0/2406404844 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7fa00c0035f0 con 0x55a4943e6660 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 10 client.16155 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 10 client.16155 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 20 client.16155 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:45.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943cdea0 0x55a4947dc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 4 client.16155 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a49472f1e0 con 0x55a4943cdea0 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa01aa00640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943cdea0 0x55a4947dc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa01aa00640 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943cdea0 0x55a4947dc3d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=117bdd3c70043c93 server_cookie=5bc1ea80410eef5d in_seq=0 out_seq=0 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 10 client.16155 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 1 -- 172.21.15.39:0/2406404844 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a49472f1e0 con 0x55a4943cdea0 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 10 client.16155 handle_command_reply: tid=0 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 client.16155 shutdown 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 msgr2=0x55a4947cac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947cac40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009a90 tx=0x7fa00c009ac0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 shutdown_connections 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943cdea0 0x55a4947dc3d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a49440f520 0x55a4947d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943ee840 0x55a4947cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 --2- 172.21.15.39:0/2406404844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947cac40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 >> 172.21.15.39:0/2406404844 conn(0x55a494403910 msgr2=0x55a4947b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:45.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa005600640 0 client.16155 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:45.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 shutdown_connections 2024-09-21T01:18:45.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 1 -- 172.21.15.39:0/2406404844 wait complete. 2024-09-21T01:18:45.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.746+0000 7fa0291bf740 20 client.16155 trim_cache size 0 max 16384 2024-09-21T01:18:45.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 14419 13.19 MB/sec warmup 72 sec latency 18691.395 ms 2024-09-21T01:18:45.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 Processor -- start 2024-09-21T01:18:45.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 -- start start 2024-09-21T01:18:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947b99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1b1570 con 0x55a4943e6660 2024-09-21T01:18:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1aedb0 con 0x55a4943ee840 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa0291bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1acc20 con 0x55a49440f520 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34804/0 (socket says 172.21.15.39:34804) 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01b400640 1 -- 172.21.15.39:0/1940969089 learned_addr learned my addr 172.21.15.39:0/1940969089 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947b99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01be00640 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01aa00640 1 -- 172.21.15.39:0/1940969089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 msgr2=0x55a4947cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01aa00640 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01aa00640 1 -- 172.21.15.39:0/1940969089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 msgr2=0x55a4947c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01aa00640 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01aa00640 1 -- 172.21.15.39:0/1940969089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4947cbd00 con 0x55a4943ee840 2024-09-21T01:18:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.768+0000 7fa01b400640 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:18:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01aa00640 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947b99d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0081050c0 tx=0x7fa00803b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26cd1bde2c8dd6a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa007e00640 1 -- 172.21.15.39:0/1940969089 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008049040 con 0x55a4943ee840 2024-09-21T01:18:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa007e00640 1 -- 172.21.15.39:0/1940969089 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa008031d00 con 0x55a4943ee840 2024-09-21T01:18:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa007e00640 1 -- 172.21.15.39:0/1940969089 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0080d71e0 con 0x55a4943ee840 2024-09-21T01:18:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/1940969089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 msgr2=0x55a4947b99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947b99d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0081050c0 tx=0x7fa00803b640 comp rx=0 tx=0).stop 2024-09-21T01:18:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/1940969089 shutdown_connections 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a49440f520 0x55a4947cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947b99d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- 172.21.15.39:0/1940969089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4943e6660 0x55a4947c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/1940969089 >> 172.21.15.39:0/1940969089 conn(0x55a494403910 msgr2=0x55a4947b92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/1940969089 shutdown_connections 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/1940969089 wait complete. 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- start start 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 Processor -- start 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 0x55a4947d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947d4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a49440f520 0x55a4947d5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1ad270 con 0x55a49440f520 2024-09-21T01:18:45.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1ae6e0 con 0x55a4943ee840 2024-09-21T01:18:45.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1ad940 con 0x55a4943e6660 2024-09-21T01:18:45.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 10 client.0 _ll_register_callbacks cb 0x7fa00c0056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:18:45.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 0x55a4947d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947d4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 0x55a4947d45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44028/0 (socket says 172.21.15.39:44028) 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 -- 172.21.15.39:0/3334583802 learned_addr learned my addr 172.21.15.39:0/3334583802 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01be00640 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a49440f520 0x55a4947d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 -- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 msgr2=0x55a4947d4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947d4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 -- 172.21.15.39:0/3334583802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a49440f520 msgr2=0x55a4947d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a49440f520 0x55a4947d5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 -- 172.21.15.39:0/3334583802 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a494845cd0 con 0x55a4943e6660 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa01b400640 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 0x55a4947d45f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa00c003bb0 tx=0x7fa00c00e700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2cda63f0b87a385 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa005600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:18:45.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa005600640 1 -- 172.21.15.39:0/3334583802 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c029040 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a49472ac90 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa005600640 1 -- 172.21.15.39:0/3334583802 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9511+0+0 (secure 0 0 0) 0x7fa00c003e30 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa005600640 1 -- 172.21.15.39:0/3334583802 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fa00c01e040 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 10 client.26182 fetch_fsmap learned FSMap version 50 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55a494846ec0 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.770+0000 7fa005600640 1 -- 172.21.15.39:0/3334583802 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a49472ac90 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa005600640 1 -- 172.21.15.39:0/3334583802 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 50) ==== 8374+0+0 (secure 0 0 0) 0x7fa00c00eb70 con 0x55a4943e6660 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa0291bf740 10 client.26182 fetch_fsmap finished waiting for FSMap version 50 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa0291bf740 10 client.26182 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa0291bf740 20 client.26182 populate_metadata read hostname 'smithi039' 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa0291bf740 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943a3250 0x55a494849580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa0291bf740 4 client.26182 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6b596caf", "release": true, "await": true}] 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6b596caf", "release": true, "await": true}) -- 0x55a4947280c0 con 0x55a4943a3250 2024-09-21T01:18:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa01aa00640 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943a3250 0x55a494849580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:18:45.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa01aa00640 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943a3250 0x55a494849580 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=829b978c5f88c3fb server_cookie=698e11a3361936a2 in_seq=0 out_seq=0 2024-09-21T01:18:45.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.775+0000 7fa005600640 10 client.26182 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:45.788 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:45 smithi039 ceph-mon[30415]: pgmap v864: 129 pgs: 129 active+clean; 828 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 1.9 MiB/s wr, 54 op/s 2024-09-21T01:18:45.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa005600640 1 -- 172.21.15.39:0/3334583802 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a4947280c0 con 0x55a4943a3250 2024-09-21T01:18:45.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa005600640 10 client.26182 handle_command_reply: tid=0 2024-09-21T01:18:45.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 client.26182 shutdown 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 msgr2=0x55a4947d45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 0x55a4947d45f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa00c003bb0 tx=0x7fa00c00e700 comp rx=0 tx=0).stop 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 shutdown_connections 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55a4943a3250 0x55a494849580 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a49440f520 0x55a4947d5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a4943ee840 0x55a4947d4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 --2- 172.21.15.39:0/3334583802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55a4943e6660 0x55a4947d45f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 >> 172.21.15.39:0/3334583802 conn(0x55a494403910 msgr2=0x55a4947b71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.804+0000 7fa005600640 0 client.26182 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:18:45.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.807+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 shutdown_connections 2024-09-21T01:18:45.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.807+0000 7fa0291bf740 1 -- 172.21.15.39:0/3334583802 wait complete. 2024-09-21T01:18:45.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.807+0000 7fa0291bf740 20 client.26182 trim_cache size 0 max 16384 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 50, 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 60, 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "6b596caf": { 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 60, 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:18:45.810 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 138.8, 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:45.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0000b6680 msgr2=0x7fa0000b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0000b6680 0x7fa0000b8b90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa014005fd0 tx=0x7fa014009500 comp rx=0 tx=0).stop 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 msgr2=0x7fa01c075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:18:45.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c075c50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa008009230 tx=0x7fa008003b90 comp rx=0 tx=0).stop 2024-09-21T01:18:45.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 shutdown_connections 2024-09-21T01:18:45.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa01c109df0 0x7fa01c076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa0000b6680 0x7fa0000b8b90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c1094a0 0x7fa01c075c50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 --2- 172.21.15.39:0/664604939 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa01c103b60 0x7fa01c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:18:45.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.811+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 >> 172.21.15.39:0/664604939 conn(0x7fa01c07b850 msgr2=0x7fa01c1074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:18:45.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.812+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 shutdown_connections 2024-09-21T01:18:45.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:18:45.812+0000 7fa022400640 1 -- 172.21.15.39:0/664604939 wait complete. 2024-09-21T01:18:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:45 smithi150 ceph-mon[34945]: pgmap v864: 129 pgs: 129 active+clean; 828 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 1.9 MiB/s wr, 54 op/s 2024-09-21T01:18:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:45 smithi160 ceph-mon[34696]: pgmap v864: 129 pgs: 129 active+clean; 828 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 1.9 MiB/s wr, 54 op/s 2024-09-21T01:18:46.684 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6b596caf', total seconds elapsed: 24.0 2024-09-21T01:18:46.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 15134 13.60 MB/sec warmup 73 sec latency 18858.284 ms 2024-09-21T01:18:47.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 15920 13.87 MB/sec warmup 74 sec latency 78.032 ms 2024-09-21T01:18:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:47 smithi039 ceph-mon[30415]: pgmap v865: 129 pgs: 129 active+clean; 830 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 2.2 MiB/s wr, 61 op/s 2024-09-21T01:18:47.908 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:47 smithi160 ceph-mon[34696]: pgmap v865: 129 pgs: 129 active+clean; 830 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 2.2 MiB/s wr, 61 op/s 2024-09-21T01:18:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:47 smithi150 ceph-mon[34945]: pgmap v865: 129 pgs: 129 active+clean; 830 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 2.2 MiB/s wr, 61 op/s 2024-09-21T01:18:48.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16903 14.54 MB/sec warmup 75 sec latency 35.338 ms 2024-09-21T01:18:49.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 17602 14.82 MB/sec warmup 76 sec latency 68.901 ms 2024-09-21T01:18:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:49 smithi150 ceph-mon[34945]: pgmap v866: 129 pgs: 129 active+clean; 840 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-09-21T01:18:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:49 smithi160 ceph-mon[34696]: pgmap v866: 129 pgs: 129 active+clean; 840 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-09-21T01:18:50.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:49 smithi039 ceph-mon[30415]: pgmap v866: 129 pgs: 129 active+clean; 840 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-09-21T01:18:50.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 18263 15.01 MB/sec warmup 77 sec latency 136.535 ms 2024-09-21T01:18:51.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 19252 15.39 MB/sec warmup 78 sec latency 57.002 ms 2024-09-21T01:18:52.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:51 smithi039 ceph-mon[30415]: pgmap v867: 129 pgs: 129 active+clean; 845 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-09-21T01:18:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:51 smithi150 ceph-mon[34945]: pgmap v867: 129 pgs: 129 active+clean; 845 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-09-21T01:18:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:51 smithi160 ceph-mon[34696]: pgmap v867: 129 pgs: 129 active+clean; 845 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-09-21T01:18:52.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 20607 16.31 MB/sec warmup 79 sec latency 33.587 ms 2024-09-21T01:18:53.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 21384 16.67 MB/sec warmup 80 sec latency 283.910 ms 2024-09-21T01:18:53.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:53 smithi039 ceph-mon[30415]: pgmap v868: 129 pgs: 129 active+clean; 848 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.3 MiB/s wr, 270 op/s 2024-09-21T01:18:54.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:53 smithi150 ceph-mon[34945]: pgmap v868: 129 pgs: 129 active+clean; 848 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.3 MiB/s wr, 270 op/s 2024-09-21T01:18:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:53 smithi160 ceph-mon[34696]: pgmap v868: 129 pgs: 129 active+clean; 848 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.3 MiB/s wr, 270 op/s 2024-09-21T01:18:54.755 INFO:tasks.workunit.client.0.smithi039.stdout: 10 22060 16.84 MB/sec warmup 81 sec latency 1283.970 ms 2024-09-21T01:18:55.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 23084 17.27 MB/sec warmup 82 sec latency 2284.019 ms 2024-09-21T01:18:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:55 smithi150 ceph-mon[34945]: pgmap v869: 129 pgs: 129 active+clean; 866 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 423 op/s 2024-09-21T01:18:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:55 smithi160 ceph-mon[34696]: pgmap v869: 129 pgs: 129 active+clean; 866 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 423 op/s 2024-09-21T01:18:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:55 smithi039 ceph-mon[30415]: pgmap v869: 129 pgs: 129 active+clean; 866 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 423 op/s 2024-09-21T01:18:56.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 24131 17.75 MB/sec warmup 83 sec latency 3284.069 ms 2024-09-21T01:18:57.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 24842 18.05 MB/sec warmup 84 sec latency 4284.130 ms 2024-09-21T01:18:57.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:57 smithi039 ceph-mon[30415]: pgmap v870: 129 pgs: 129 active+clean; 871 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.2 MiB/s wr, 438 op/s 2024-09-21T01:18:58.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:57 smithi150 ceph-mon[34945]: pgmap v870: 129 pgs: 129 active+clean; 871 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.2 MiB/s wr, 438 op/s 2024-09-21T01:18:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:57 smithi160 ceph-mon[34696]: pgmap v870: 129 pgs: 129 active+clean; 871 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.2 MiB/s wr, 438 op/s 2024-09-21T01:18:58.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 25657 18.25 MB/sec warmup 85 sec latency 5284.192 ms 2024-09-21T01:18:59.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 26364 18.55 MB/sec warmup 86 sec latency 6092.186 ms 2024-09-21T01:18:59.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:18:59 smithi039 ceph-mon[30415]: pgmap v871: 129 pgs: 129 active+clean; 894 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.1 MiB/s wr, 579 op/s 2024-09-21T01:19:00.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:18:59 smithi150 ceph-mon[34945]: pgmap v871: 129 pgs: 129 active+clean; 894 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.1 MiB/s wr, 579 op/s 2024-09-21T01:19:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:18:59 smithi160 ceph-mon[34696]: pgmap v871: 129 pgs: 129 active+clean; 894 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.1 MiB/s wr, 579 op/s 2024-09-21T01:19:00.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 27383 18.96 MB/sec warmup 87 sec latency 57.579 ms 2024-09-21T01:19:01.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 27890 19.07 MB/sec warmup 88 sec latency 290.142 ms 2024-09-21T01:19:01.842 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:01 smithi039 ceph-mon[30415]: pgmap v872: 129 pgs: 129 active+clean; 901 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 534 op/s 2024-09-21T01:19:02.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:01 smithi150 ceph-mon[34945]: pgmap v872: 129 pgs: 129 active+clean; 901 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 534 op/s 2024-09-21T01:19:02.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:01 smithi160 ceph-mon[34696]: pgmap v872: 129 pgs: 129 active+clean; 901 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 534 op/s 2024-09-21T01:19:02.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 27893 18.86 MB/sec warmup 89 sec latency 1290.212 ms 2024-09-21T01:19:03.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 27893 18.65 MB/sec warmup 90 sec latency 2290.280 ms 2024-09-21T01:19:03.853 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:03 smithi039 ceph-mon[30415]: pgmap v873: 129 pgs: 129 active+clean; 919 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.1 MiB/s wr, 544 op/s 2024-09-21T01:19:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:03 smithi150 ceph-mon[34945]: pgmap v873: 129 pgs: 129 active+clean; 919 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.1 MiB/s wr, 544 op/s 2024-09-21T01:19:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:03 smithi160 ceph-mon[34696]: pgmap v873: 129 pgs: 129 active+clean; 919 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.1 MiB/s wr, 544 op/s 2024-09-21T01:19:04.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 27895 18.45 MB/sec warmup 91 sec latency 3095.223 ms 2024-09-21T01:19:05.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28042 18.36 MB/sec warmup 92 sec latency 3935.177 ms 2024-09-21T01:19:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:05 smithi039 ceph-mon[30415]: pgmap v874: 129 pgs: 129 active+clean; 944 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 605 op/s 2024-09-21T01:19:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:05 smithi039 ceph-mon[30415]: osdmap e109: 12 total, 12 up, 12 in 2024-09-21T01:19:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:05 smithi150 ceph-mon[34945]: pgmap v874: 129 pgs: 129 active+clean; 944 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 605 op/s 2024-09-21T01:19:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:05 smithi150 ceph-mon[34945]: osdmap e109: 12 total, 12 up, 12 in 2024-09-21T01:19:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:05 smithi160 ceph-mon[34696]: pgmap v874: 129 pgs: 129 active+clean; 944 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 605 op/s 2024-09-21T01:19:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:05 smithi160 ceph-mon[34696]: osdmap e109: 12 total, 12 up, 12 in 2024-09-21T01:19:06.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28107 18.18 MB/sec warmup 93 sec latency 4587.335 ms 2024-09-21T01:19:07.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28120 17.99 MB/sec warmup 94 sec latency 5587.403 ms 2024-09-21T01:19:07.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:07 smithi039 ceph-mon[30415]: pgmap v876: 129 pgs: 129 active+clean; 952 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 10 MiB/s wr, 536 op/s 2024-09-21T01:19:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:07 smithi150 ceph-mon[34945]: pgmap v876: 129 pgs: 129 active+clean; 952 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 10 MiB/s wr, 536 op/s 2024-09-21T01:19:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:07 smithi160 ceph-mon[34696]: pgmap v876: 129 pgs: 129 active+clean; 952 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 10 MiB/s wr, 536 op/s 2024-09-21T01:19:08.756 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28123 17.80 MB/sec warmup 95 sec latency 6298.765 ms 2024-09-21T01:19:09.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28125 17.61 MB/sec warmup 96 sec latency 6922.204 ms 2024-09-21T01:19:09.947 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:09 smithi039 ceph-mon[30415]: pgmap v877: 129 pgs: 129 active+clean; 981 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 498 op/s 2024-09-21T01:19:10.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:09 smithi150 ceph-mon[34945]: pgmap v877: 129 pgs: 129 active+clean; 981 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 498 op/s 2024-09-21T01:19:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:09 smithi160 ceph-mon[34696]: pgmap v877: 129 pgs: 129 active+clean; 981 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 498 op/s 2024-09-21T01:19:10.758 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28126 17.43 MB/sec warmup 97 sec latency 3565.693 ms 2024-09-21T01:19:11.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28179 17.30 MB/sec warmup 98 sec latency 4024.946 ms 2024-09-21T01:19:12.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:11 smithi150 ceph-mon[34945]: pgmap v878: 129 pgs: 129 active+clean; 988 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 13 MiB/s wr, 482 op/s 2024-09-21T01:19:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:11 smithi160 ceph-mon[34696]: pgmap v878: 129 pgs: 129 active+clean; 988 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 13 MiB/s wr, 482 op/s 2024-09-21T01:19:12.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:11 smithi039 ceph-mon[30415]: pgmap v878: 129 pgs: 129 active+clean; 988 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 13 MiB/s wr, 482 op/s 2024-09-21T01:19:12.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28377 17.27 MB/sec warmup 99 sec latency 4336.394 ms 2024-09-21T01:19:13.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:12 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:19:13.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:12 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:19:13.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:12 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:19:13.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28462 17.10 MB/sec warmup 100 sec latency 5336.443 ms 2024-09-21T01:19:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:13 smithi150 ceph-mon[34945]: pgmap v879: 129 pgs: 129 active+clean; 982 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 13 MiB/s wr, 448 op/s 2024-09-21T01:19:14.111 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:13 smithi160 ceph-mon[34696]: pgmap v879: 129 pgs: 129 active+clean; 982 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 13 MiB/s wr, 448 op/s 2024-09-21T01:19:14.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:13 smithi039 ceph-mon[30415]: pgmap v879: 129 pgs: 129 active+clean; 982 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 13 MiB/s wr, 448 op/s 2024-09-21T01:19:14.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.98 MB/sec warmup 101 sec latency 6336.496 ms 2024-09-21T01:19:15.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.81 MB/sec warmup 102 sec latency 7336.542 ms 2024-09-21T01:19:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:15 smithi150 ceph-mon[34945]: pgmap v880: 129 pgs: 129 active+clean; 956 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 398 op/s 2024-09-21T01:19:15.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:15 smithi150 ceph-mon[34945]: osdmap e110: 12 total, 12 up, 12 in 2024-09-21T01:19:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:15 smithi160 ceph-mon[34696]: pgmap v880: 129 pgs: 129 active+clean; 956 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 398 op/s 2024-09-21T01:19:15.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:15 smithi160 ceph-mon[34696]: osdmap e110: 12 total, 12 up, 12 in 2024-09-21T01:19:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:15 smithi039 ceph-mon[30415]: pgmap v880: 129 pgs: 129 active+clean; 956 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 398 op/s 2024-09-21T01:19:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:15 smithi039 ceph-mon[30415]: osdmap e110: 12 total, 12 up, 12 in 2024-09-21T01:19:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.65 MB/sec warmup 103 sec latency 8336.586 ms 2024-09-21T01:19:17.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.49 MB/sec warmup 104 sec latency 9336.631 ms 2024-09-21T01:19:17.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:17 smithi150 ceph-mon[34945]: pgmap v882: 129 pgs: 129 active+clean; 950 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 10 MiB/s wr, 368 op/s 2024-09-21T01:19:17.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:17 smithi160 ceph-mon[34696]: pgmap v882: 129 pgs: 129 active+clean; 950 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 10 MiB/s wr, 368 op/s 2024-09-21T01:19:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:17 smithi039 ceph-mon[30415]: pgmap v882: 129 pgs: 129 active+clean; 950 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 10 MiB/s wr, 368 op/s 2024-09-21T01:19:18.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.33 MB/sec warmup 105 sec latency 10336.675 ms 2024-09-21T01:19:19.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.18 MB/sec warmup 106 sec latency 11336.722 ms 2024-09-21T01:19:19.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:19 smithi150 ceph-mon[34945]: pgmap v883: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 264 op/s 2024-09-21T01:19:19.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:19 smithi160 ceph-mon[34696]: pgmap v883: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 264 op/s 2024-09-21T01:19:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:19 smithi039 ceph-mon[30415]: pgmap v883: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 264 op/s 2024-09-21T01:19:20.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 16.02 MB/sec warmup 107 sec latency 12336.767 ms 2024-09-21T01:19:21.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 15.88 MB/sec warmup 108 sec latency 13336.830 ms 2024-09-21T01:19:21.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:21 smithi150 ceph-mon[34945]: pgmap v884: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 225 op/s 2024-09-21T01:19:21.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:21 smithi160 ceph-mon[34696]: pgmap v884: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 225 op/s 2024-09-21T01:19:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:21 smithi039 ceph-mon[30415]: pgmap v884: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 225 op/s 2024-09-21T01:19:22.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28501 15.73 MB/sec warmup 109 sec latency 14336.893 ms 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: pgmap v885: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:19:23.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:19:23.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28540 15.62 MB/sec warmup 110 sec latency 15336.958 ms 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: pgmap v885: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:19:23.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:19:23.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:19:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: pgmap v885: 129 pgs: 129 active+clean; 946 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:19:23.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:19:24.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 28540 15.48 MB/sec warmup 111 sec latency 16337.022 ms 2024-09-21T01:19:24.820 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:24 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:19:24.820 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:24 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:19:24.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:24 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:19:24.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:24 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:19:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:24 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:19:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:24 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:19:25.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 29129 15.49 MB/sec warmup 112 sec latency 16769.251 ms 2024-09-21T01:19:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:25 smithi160 ceph-mon[34696]: pgmap v886: 129 pgs: 129 active+clean; 947 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 739 KiB/s wr, 73 op/s 2024-09-21T01:19:25.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:25 smithi039 ceph-mon[30415]: pgmap v886: 129 pgs: 129 active+clean; 947 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 739 KiB/s wr, 73 op/s 2024-09-21T01:19:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:25 smithi150 ceph-mon[34945]: pgmap v886: 129 pgs: 129 active+clean; 947 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 739 KiB/s wr, 73 op/s 2024-09-21T01:19:26.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 30212 15.95 MB/sec warmup 113 sec latency 61.435 ms 2024-09-21T01:19:27.760 INFO:tasks.workunit.client.0.smithi039.stdout: 10 30939 16.20 MB/sec warmup 114 sec latency 67.651 ms 2024-09-21T01:19:27.830 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:27 smithi039 ceph-mon[30415]: pgmap v887: 129 pgs: 129 active+clean; 948 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 698 KiB/s wr, 64 op/s 2024-09-21T01:19:27.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:27 smithi160 ceph-mon[34696]: pgmap v887: 129 pgs: 129 active+clean; 948 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 698 KiB/s wr, 64 op/s 2024-09-21T01:19:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:27 smithi150 ceph-mon[34945]: pgmap v887: 129 pgs: 129 active+clean; 948 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 698 KiB/s wr, 64 op/s 2024-09-21T01:19:28.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 31528 16.23 MB/sec warmup 115 sec latency 69.094 ms 2024-09-21T01:19:29.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 32608 16.53 MB/sec warmup 116 sec latency 44.209 ms 2024-09-21T01:19:29.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:29 smithi160 ceph-mon[34696]: pgmap v888: 129 pgs: 129 active+clean; 945 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 106 op/s 2024-09-21T01:19:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:29 smithi039 ceph-mon[30415]: pgmap v888: 129 pgs: 129 active+clean; 945 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 106 op/s 2024-09-21T01:19:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:29 smithi150 ceph-mon[34945]: pgmap v888: 129 pgs: 129 active+clean; 945 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 106 op/s 2024-09-21T01:19:30.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 33771 17.06 MB/sec warmup 117 sec latency 40.615 ms 2024-09-21T01:19:31.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 34612 17.30 MB/sec warmup 118 sec latency 145.978 ms 2024-09-21T01:19:31.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:31 smithi160 ceph-mon[34696]: pgmap v889: 129 pgs: 129 active+clean; 949 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 190 op/s 2024-09-21T01:19:31.881 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:31 smithi039 ceph-mon[30415]: pgmap v889: 129 pgs: 129 active+clean; 949 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 190 op/s 2024-09-21T01:19:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:31 smithi150 ceph-mon[34945]: pgmap v889: 129 pgs: 129 active+clean; 949 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 190 op/s 2024-09-21T01:19:32.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 35222 17.43 MB/sec warmup 119 sec latency 125.308 ms 2024-09-21T01:19:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:33 smithi160 ceph-mon[34696]: pgmap v890: 129 pgs: 129 active+clean; 958 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.7 MiB/s wr, 248 op/s 2024-09-21T01:19:33.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:33 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:19:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:33 smithi039 ceph-mon[30415]: pgmap v890: 129 pgs: 129 active+clean; 958 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.7 MiB/s wr, 248 op/s 2024-09-21T01:19:33.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:33 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:19:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:33 smithi150 ceph-mon[34945]: pgmap v890: 129 pgs: 129 active+clean; 958 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.7 MiB/s wr, 248 op/s 2024-09-21T01:19:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:33 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:19:34.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 37315 79.96 MB/sec execute 1 sec latency 67.772 ms 2024-09-21T01:19:35.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 38120 60.71 MB/sec execute 2 sec latency 128.887 ms 2024-09-21T01:19:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:35 smithi039 ceph-mon[30415]: pgmap v891: 129 pgs: 129 active+clean; 968 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.7 MiB/s wr, 315 op/s 2024-09-21T01:19:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:35 smithi150 ceph-mon[34945]: pgmap v891: 129 pgs: 129 active+clean; 968 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.7 MiB/s wr, 315 op/s 2024-09-21T01:19:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:35 smithi160 ceph-mon[34696]: pgmap v891: 129 pgs: 129 active+clean; 968 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.7 MiB/s wr, 315 op/s 2024-09-21T01:19:36.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 38854 53.03 MB/sec execute 3 sec latency 57.650 ms 2024-09-21T01:19:37.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 39817 49.57 MB/sec execute 4 sec latency 55.716 ms 2024-09-21T01:19:37.872 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:37 smithi039 ceph-mon[30415]: pgmap v892: 129 pgs: 129 active+clean; 982 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 5.9 MiB/s wr, 394 op/s 2024-09-21T01:19:38.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:37 smithi150 ceph-mon[34945]: pgmap v892: 129 pgs: 129 active+clean; 982 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 5.9 MiB/s wr, 394 op/s 2024-09-21T01:19:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:37 smithi160 ceph-mon[34696]: pgmap v892: 129 pgs: 129 active+clean; 982 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 5.9 MiB/s wr, 394 op/s 2024-09-21T01:19:38.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 40444 48.24 MB/sec execute 5 sec latency 256.533 ms 2024-09-21T01:19:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:39 smithi039 ceph-mon[30415]: pgmap v893: 129 pgs: 129 active+clean; 1009 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 545 op/s 2024-09-21T01:19:39.761 INFO:tasks.workunit.client.0.smithi039.stdout: 10 41280 49.29 MB/sec execute 6 sec latency 74.866 ms 2024-09-21T01:19:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:39 smithi150 ceph-mon[34945]: pgmap v893: 129 pgs: 129 active+clean; 1009 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 545 op/s 2024-09-21T01:19:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:39 smithi160 ceph-mon[34696]: pgmap v893: 129 pgs: 129 active+clean; 1009 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 545 op/s 2024-09-21T01:19:40.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 41855 46.34 MB/sec execute 7 sec latency 106.710 ms 2024-09-21T01:19:41.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 42619 44.29 MB/sec execute 8 sec latency 53.470 ms 2024-09-21T01:19:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:41 smithi039 ceph-mon[30415]: pgmap v894: 129 pgs: 129 active+clean; 1018 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.5 MiB/s wr, 545 op/s 2024-09-21T01:19:42.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:41 smithi150 ceph-mon[34945]: pgmap v894: 129 pgs: 129 active+clean; 1018 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.5 MiB/s wr, 545 op/s 2024-09-21T01:19:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:41 smithi160 ceph-mon[34696]: pgmap v894: 129 pgs: 129 active+clean; 1018 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.5 MiB/s wr, 545 op/s 2024-09-21T01:19:42.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 43748 46.50 MB/sec execute 9 sec latency 50.902 ms 2024-09-21T01:19:43.753 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:43 smithi039 ceph-mon[30415]: pgmap v895: 129 pgs: 129 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 7.6 MiB/s wr, 482 op/s 2024-09-21T01:19:43.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 43956 42.91 MB/sec execute 10 sec latency 844.320 ms 2024-09-21T01:19:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:43 smithi150 ceph-mon[34945]: pgmap v895: 129 pgs: 129 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 7.6 MiB/s wr, 482 op/s 2024-09-21T01:19:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:43 smithi160 ceph-mon[34696]: pgmap v895: 129 pgs: 129 active+clean; 1.0 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 7.6 MiB/s wr, 482 op/s 2024-09-21T01:19:44.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 43957 39.01 MB/sec execute 11 sec latency 1844.378 ms 2024-09-21T01:19:45.713 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:45 smithi039 ceph-mon[30415]: pgmap v896: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 547 op/s 2024-09-21T01:19:45.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 43977 36.01 MB/sec execute 12 sec latency 2844.431 ms 2024-09-21T01:19:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:45 smithi150 ceph-mon[34945]: pgmap v896: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 547 op/s 2024-09-21T01:19:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:45 smithi160 ceph-mon[34696]: pgmap v896: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 547 op/s 2024-09-21T01:19:46.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44029 33.65 MB/sec execute 13 sec latency 3844.473 ms 2024-09-21T01:19:47.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44029 31.25 MB/sec execute 14 sec latency 4844.522 ms 2024-09-21T01:19:47.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:47 smithi039 ceph-mon[30415]: pgmap v897: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.6 MiB/s wr, 522 op/s 2024-09-21T01:19:47.909 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:47 smithi160 ceph-mon[34696]: pgmap v897: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.6 MiB/s wr, 522 op/s 2024-09-21T01:19:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:47 smithi150 ceph-mon[34945]: pgmap v897: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.6 MiB/s wr, 522 op/s 2024-09-21T01:19:48.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44030 29.17 MB/sec execute 15 sec latency 5620.592 ms 2024-09-21T01:19:49.733 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:49 smithi039 ceph-mon[30415]: pgmap v898: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 10 MiB/s wr, 555 op/s 2024-09-21T01:19:49.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44031 27.35 MB/sec execute 16 sec latency 6235.024 ms 2024-09-21T01:19:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:49 smithi150 ceph-mon[34945]: pgmap v898: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 10 MiB/s wr, 555 op/s 2024-09-21T01:19:50.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:49 smithi160 ceph-mon[34696]: pgmap v898: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 10 MiB/s wr, 555 op/s 2024-09-21T01:19:50.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44031 25.74 MB/sec execute 17 sec latency 7235.067 ms 2024-09-21T01:19:51.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 24.31 MB/sec execute 18 sec latency 8235.122 ms 2024-09-21T01:19:51.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:51 smithi039 ceph-mon[30415]: pgmap v899: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.2 MiB/s wr, 433 op/s 2024-09-21T01:19:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:51 smithi150 ceph-mon[34945]: pgmap v899: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.2 MiB/s wr, 433 op/s 2024-09-21T01:19:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:51 smithi160 ceph-mon[34696]: pgmap v899: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.2 MiB/s wr, 433 op/s 2024-09-21T01:19:52.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 23.03 MB/sec execute 19 sec latency 9235.191 ms 2024-09-21T01:19:53.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 21.88 MB/sec execute 20 sec latency 10235.236 ms 2024-09-21T01:19:53.912 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:53 smithi039 ceph-mon[30415]: pgmap v900: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 435 op/s 2024-09-21T01:19:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:53 smithi150 ceph-mon[34945]: pgmap v900: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 435 op/s 2024-09-21T01:19:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:53 smithi160 ceph-mon[34696]: pgmap v900: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 11 MiB/s wr, 435 op/s 2024-09-21T01:19:54.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 20.84 MB/sec execute 21 sec latency 11235.279 ms 2024-09-21T01:19:55.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 19.89 MB/sec execute 22 sec latency 12235.345 ms 2024-09-21T01:19:55.961 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:55 smithi039 ceph-mon[30415]: pgmap v901: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 14 MiB/s wr, 478 op/s 2024-09-21T01:19:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:55 smithi150 ceph-mon[34945]: pgmap v901: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 14 MiB/s wr, 478 op/s 2024-09-21T01:19:56.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:55 smithi160 ceph-mon[34696]: pgmap v901: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 14 MiB/s wr, 478 op/s 2024-09-21T01:19:56.699 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.2 2024-09-21T01:19:56.700 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:19:56.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 19.03 MB/sec execute 23 sec latency 13235.388 ms 2024-09-21T01:19:57.098 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:19:57.762 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 18.23 MB/sec execute 24 sec latency 14235.454 ms 2024-09-21T01:19:58.022 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:57 smithi039 ceph-mon[30415]: pgmap v902: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 359 op/s 2024-09-21T01:19:58.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:57 smithi150 ceph-mon[34945]: pgmap v902: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 359 op/s 2024-09-21T01:19:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:57 smithi160 ceph-mon[34696]: pgmap v902: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 359 op/s 2024-09-21T01:19:58.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 17.50 MB/sec execute 25 sec latency 15235.500 ms 2024-09-21T01:19:58.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/456123036 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d100757b0 msgr2=0x7f6d10075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:19:58.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/456123036 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d100757b0 0x7f6d10075bc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009900 tx=0x7f6d0002a420 comp rx=0 tx=0).stop 2024-09-21T01:19:58.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/456123036 shutdown_connections 2024-09-21T01:19:58.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/456123036 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d1010ff10 0x7f6d10112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:58.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/456123036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d10076100 0x7f6d1010f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:58.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/456123036 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d100757b0 0x7f6d10075bc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:58.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/456123036 >> 172.21.15.39:0/456123036 conn(0x7f6d100ff440 msgr2=0x7f6d10101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:19:58.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/456123036 shutdown_connections 2024-09-21T01:19:58.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/456123036 wait complete. 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 Processor -- start 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- start start 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d100757b0 0x7f6d1019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 0x7f6d1019cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1010ff10 0x7f6d101a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6d10115b70 con 0x7f6d10076100 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101159f0 con 0x7f6d1010ff10 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6d10115cf0 con 0x7f6d100757b0 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 0x7f6d1019cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 0x7f6d1019cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49122/0 (socket says 172.21.15.39:49122) 2024-09-21T01:19:58.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 -- 172.21.15.39:0/2179408737 learned_addr learned my addr 172.21.15.39:0/2179408737 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:19:58.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 -- 172.21.15.39:0/2179408737 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d100757b0 msgr2=0x7f6d1019c560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:19:58.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d100757b0 0x7f6d1019c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:58.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 -- 172.21.15.39:0/2179408737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1010ff10 msgr2=0x7f6d101a1da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:19:58.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1010ff10 0x7f6d101a1da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:58.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 -- 172.21.15.39:0/2179408737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d101a24d0 con 0x7f6d10076100 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0e000640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 0x7f6d1019cb00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00e9c0 tx=0x7f6cfc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc00cd80 con 0x7f6d10076100 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cfc0035f0 con 0x7f6d10076100 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d101a27c0 con 0x7f6d10076100 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.893+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d101a8fa0 con 0x7f6d10076100 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.899+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc010690 con 0x7f6d10076100 2024-09-21T01:19:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.899+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf00052c0 con 0x7f6d10076100 2024-09-21T01:19:58.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.899+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6cfc004020 con 0x7f6d10076100 2024-09-21T01:19:58.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.899+0000 7f6d07400640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cf40b6700 0x7f6cf40b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:19:58.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.899+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f6cfc0e7f30 con 0x7f6d10076100 2024-09-21T01:19:58.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.899+0000 7f6d0ea00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cf40b6700 0x7f6cf40b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:19:58.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.904+0000 7f6d0ea00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cf40b6700 0x7f6cf40b8c10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6d00005f70 tx=0x7f6d00005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:19:58.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:58.905+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6cfc0ae040 con 0x7f6d10076100 2024-09-21T01:19:59.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.150+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cf0005880 con 0x7f6d10076100 2024-09-21T01:19:59.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.151+0000 7f6d07400640 1 -- 172.21.15.39:0/2179408737 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11916 (secure 0 0 0) 0x7f6cfc0b2420 con 0x7f6d10076100 2024-09-21T01:19:59.152 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:19:59.152 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"btime":"2024-09-21T01:19:32:589648+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:19:32.030226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:19:59.153 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cf40b6700 msgr2=0x7f6cf40b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cf40b6700 0x7f6cf40b8c10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6d00005f70 tx=0x7f6d00005fd0 comp rx=0 tx=0).stop 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 msgr2=0x7f6d1019cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 0x7f6d1019cb00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00e9c0 tx=0x7f6cfc00ee80 comp rx=0 tx=0).stop 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 shutdown_connections 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6cf40b6700 0x7f6cf40b8c10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1010ff10 0x7f6d101a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:59.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d10076100 0x7f6d1019cb00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:59.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 --2- 172.21.15.39:0/2179408737 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6d100757b0 0x7f6d1019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:19:59.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 >> 172.21.15.39:0/2179408737 conn(0x7f6d100ff440 msgr2=0x7f6d10107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:19:59.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 shutdown_connections 2024-09-21T01:19:59.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:19:59.153+0000 7f6d0fe00640 1 -- 172.21.15.39:0/2179408737 wait complete. 2024-09-21T01:19:59.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 16.83 MB/sec execute 26 sec latency 16235.824 ms 2024-09-21T01:19:59.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:59 smithi039 ceph-mon[30415]: pgmap v903: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 12 MiB/s wr, 328 op/s 2024-09-21T01:19:59.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:19:59 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2179408737' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:20:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:59 smithi150 ceph-mon[34945]: pgmap v903: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 12 MiB/s wr, 328 op/s 2024-09-21T01:20:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:19:59 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2179408737' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:20:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:59 smithi160 ceph-mon[34696]: pgmap v903: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 12 MiB/s wr, 328 op/s 2024-09-21T01:20:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:19:59 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2179408737' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:20:00.215 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 143.2 --await-for 45.0' 2024-09-21T01:20:00.215 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 143.2 --await-for 45.0 2024-09-21T01:20:00.756 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:20:00.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 16.21 MB/sec execute 27 sec latency 17235.872 ms 2024-09-21T01:20:00.801 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:00 smithi039 ceph-mon[30415]: overall HEALTH_OK 2024-09-21T01:20:01.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:00 smithi150 ceph-mon[34945]: overall HEALTH_OK 2024-09-21T01:20:01.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:00 smithi160 ceph-mon[34696]: overall HEALTH_OK 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- 172.21.15.39:0/2615165645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 msgr2=0x7f5fdc110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- 172.21.15.39:0/2615165645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc110140 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009ca0 tx=0x7f5fd402a8d0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- 172.21.15.39:0/2615165645 shutdown_connections 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- 172.21.15.39:0/2615165645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc110140 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- 172.21.15.39:0/2615165645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc10d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- 172.21.15.39:0/2615165645 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5fdc1047f0 0x7f5fdc104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- 172.21.15.39:0/2615165645 >> 172.21.15.39:0/2615165645 conn(0x7f5fdc0fe330 msgr2=0x7f5fdc100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- 172.21.15.39:0/2615165645 shutdown_connections 2024-09-21T01:20:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- 172.21.15.39:0/2615165645 wait complete. 2024-09-21T01:20:01.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 Processor -- start 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- start start 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5fdc1047f0 0x7f5fdc102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc102fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1139a0 con 0x7f5fdc10dc50 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc113820 con 0x7f5fdc105140 2024-09-21T01:20:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.743+0000 7f5fe1800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc113b20 con 0x7f5fdc1047f0 2024-09-21T01:20:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5fdc1047f0 0x7f5fdc102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35748/0 (socket says 172.21.15.39:35748) 2024-09-21T01:20:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 -- 172.21.15.39:0/3160984995 learned_addr learned my addr 172.21.15.39:0/3160984995 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 -- 172.21.15.39:0/3160984995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5fdc1047f0 msgr2=0x7f5fdc102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc102fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5fdc1047f0 0x7f5fdc102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 -- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 msgr2=0x7f5fdc102fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc102fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 -- 172.21.15.39:0/3160984995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc1a85d0 con 0x7f5fdc105140 2024-09-21T01:20:01.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fdb400640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc102a70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800e9f0 tx=0x7f5fc800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fd8c00640 1 -- 172.21.15.39:0/3160984995 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc800ccc0 con 0x7f5fdc105140 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fdc1a88c0 con 0x7f5fdc105140 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fd8c00640 1 -- 172.21.15.39:0/3160984995 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fc80108e0 con 0x7f5fdc105140 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fd8c00640 1 -- 172.21.15.39:0/3160984995 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8018860 con 0x7f5fdc105140 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.748+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fdc1a8e80 con 0x7f5fdc105140 2024-09-21T01:20:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 Processor -- start 2024-09-21T01:20:01.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 -- start start 2024-09-21T01:20:01.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d2e840 0x55eeda0d6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0d6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0d6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55eeda04bca0 con 0x55eed9d0db60 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55eeda03b0d0 con 0x55eed9d05980 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eeda03eee0 con 0x55eed9d2e840 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fdb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0d6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fdb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0d6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49160/0 (socket says 172.21.15.39:49160) 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fdb400640 1 -- 172.21.15.39:0/2397706192 learned_addr learned my addr 172.21.15.39:0/2397706192 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.753+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d2e840 0x55eeda0d6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fe0e00640 1 -- 172.21.15.39:0/2397706192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 msgr2=0x55eeda0d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0d6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fe0e00640 1 -- 172.21.15.39:0/2397706192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 msgr2=0x55eeda0d6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0d6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fe0e00640 1 -- 172.21.15.39:0/2397706192 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eed9fed400 con 0x55eed9d2e840 2024-09-21T01:20:01.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fd8c00640 1 -- 172.21.15.39:0/3160984995 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5fc8010430 con 0x7f5fdc105140 2024-09-21T01:20:01.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d2e840 0x55eeda0d6510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4005e30 tx=0x7f5fd402f3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7138bbebc291d29a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:01.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fd1600640 1 -- 172.21.15.39:0/2397706192 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4031b00 con 0x55eed9d2e840 2024-09-21T01:20:01.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fd1600640 1 -- 172.21.15.39:0/2397706192 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd4036070 con 0x55eed9d2e840 2024-09-21T01:20:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.754+0000 7f5fd1600640 1 -- 172.21.15.39:0/2397706192 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4004040 con 0x55eed9d2e840 2024-09-21T01:20:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fd8c00640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5fc00b6670 0x7f5fc00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2397706192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d2e840 msgr2=0x55eeda0d6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d2e840 0x55eeda0d6510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4005e30 tx=0x7f5fd402f3b0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fd8c00640 1 -- 172.21.15.39:0/3160984995 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f5fc80eec80 con 0x7f5fdc105140 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2397706192 shutdown_connections 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0d6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0d6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2397706192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d2e840 0x55eeda0d6510 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2397706192 >> 172.21.15.39:0/2397706192 conn(0x55eed9d22990 msgr2=0x55eeda0d4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5fc00b6670 0x7f5fc00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2397706192 shutdown_connections 2024-09-21T01:20:01.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 15.63 MB/sec execute 28 sec latency 18235.932 ms 2024-09-21T01:20:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2397706192 wait complete. 2024-09-21T01:20:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- start start 2024-09-21T01:20:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 Processor -- start 2024-09-21T01:20:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5fc00b6670 0x7f5fc00b8b80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55eeda0d5190 tx=0x7f5fcc008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:01.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d05980 0x55eeda0e1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55eed9e9c7a0 con 0x55eed9d0db60 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55eed9fe0a10 con 0x55eed9d2e840 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eeda0e4c90 con 0x55eed9d05980 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe9a30740 10 client.0 _ll_register_callbacks cb 0x55eed9ff0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d05980 0x55eeda0e1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d05980 0x55eeda0e1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:48178/0 (socket says 172.21.15.39:48178) 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fdbe00640 1 -- 172.21.15.39:0/3954137535 learned_addr learned my addr 172.21.15.39:0/3954137535 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fdb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.760+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fe0e00640 1 -- 172.21.15.39:0/3954137535 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d05980 msgr2=0x55eeda0e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d05980 0x55eeda0e1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fe0e00640 1 -- 172.21.15.39:0/3954137535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 msgr2=0x55eeda0e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0e20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fe0e00640 1 -- 172.21.15.39:0/3954137535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eed9da8b90 con 0x55eed9d2e840 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0edf80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5fd402aec0 tx=0x7f5fd40039f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abd6499a53c4cc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.765+0000 7f5fc6000640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fc6000640 1 -- 172.21.15.39:0/3954137535 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4031d10 con 0x55eed9d2e840 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fc6000640 1 -- 172.21.15.39:0/3954137535 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f5fd4031460 con 0x55eed9d2e840 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fd4036070 con 0x55eed9d2e840 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fc6000640 1 -- 172.21.15.39:0/3954137535 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f5fd405b050 con 0x55eed9d2e840 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 10 client.26201 fetch_fsmap learned FSMap version 51 2024-09-21T01:20:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55eeda0f9180 con 0x55eed9d2e840 2024-09-21T01:20:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fc6000640 1 -- 172.21.15.39:0/3954137535 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd4036070 con 0x55eed9d2e840 2024-09-21T01:20:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fc6000640 1 -- 172.21.15.39:0/3954137535 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 51) ==== 8370+0+0 (secure 0 0 0) 0x7f5fd40305d0 con 0x55eed9d2e840 2024-09-21T01:20:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 10 client.26201 fetch_fsmap finished waiting for FSMap version 51 2024-09-21T01:20:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 10 client.26201 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:20:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 20 client.26201 populate_metadata read hostname 'smithi039' 2024-09-21T01:20:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9ced1c0 0x55eeda0fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 4 client.26201 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:20:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eeda04dbf0 con 0x55eed9ced1c0 2024-09-21T01:20:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.766+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9ced1c0 0x55eeda0fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9ced1c0 0x55eeda0fc4e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58886dd867a6eb7e server_cookie=b7ac3d9cbb53effa in_seq=0 out_seq=0 2024-09-21T01:20:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fc6000640 10 client.26201 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fc6000640 1 -- 172.21.15.39:0/3954137535 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55eeda04dbf0 con 0x55eed9ced1c0 2024-09-21T01:20:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fc6000640 10 client.26201 handle_command_reply: tid=0 2024-09-21T01:20:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 client.26201 shutdown 2024-09-21T01:20:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 msgr2=0x55eeda0edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0edf80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5fd402aec0 tx=0x7f5fd40039f0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 shutdown_connections 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9ced1c0 0x55eeda0fc4e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fc6000640 0 client.26201 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0edf80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d0db60 0x55eeda0e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3954137535 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d05980 0x55eeda0e1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 >> 172.21.15.39:0/3954137535 conn(0x55eed9d22990 msgr2=0x55eeda0d96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 shutdown_connections 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3954137535 wait complete. 2024-09-21T01:20:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.778+0000 7f5fe9a30740 20 client.26201 trim_cache size 0 max 16384 2024-09-21T01:20:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 Processor -- start 2024-09-21T01:20:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 -- start start 2024-09-21T01:20:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 0x55eeda0e3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0dde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1b15e0 con 0x55eed9d05980 2024-09-21T01:20:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1aee50 con 0x55eed9d2e840 2024-09-21T01:20:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.812+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1accc0 con 0x55eed9d0db60 2024-09-21T01:20:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 0x55eeda0e3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 0x55eeda0e3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49180/0 (socket says 172.21.15.39:49180) 2024-09-21T01:20:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 -- 172.21.15.39:0/2345711201 learned_addr learned my addr 172.21.15.39:0/2345711201 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdb400640 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 -- 172.21.15.39:0/2345711201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 msgr2=0x55eeda0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 -- 172.21.15.39:0/2345711201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 msgr2=0x55eeda0dde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0dde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 -- 172.21.15.39:0/2345711201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeda0da620 con 0x55eed9d05980 2024-09-21T01:20:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 0x55eeda0e3a80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55eeda0e43f0 tx=0x7f5fcc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac8421197b56e0d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fd0c00640 1 -- 172.21.15.39:0/2345711201 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0228e0 con 0x55eed9d05980 2024-09-21T01:20:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fd0c00640 1 -- 172.21.15.39:0/2345711201 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fcc02e070 con 0x55eed9d05980 2024-09-21T01:20:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fd0c00640 1 -- 172.21.15.39:0/2345711201 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc029c70 con 0x55eed9d05980 2024-09-21T01:20:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2345711201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 msgr2=0x55eeda0e3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 0x55eeda0e3a80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55eeda0e43f0 tx=0x7f5fcc010000 comp rx=0 tx=0).stop 2024-09-21T01:20:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2345711201 shutdown_connections 2024-09-21T01:20:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d2e840 0x55eeda0dde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/2345711201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d05980 0x55eeda0e3a80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2345711201 >> 172.21.15.39:0/2345711201 conn(0x55eed9d22990 msgr2=0x55eeda0d74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2345711201 shutdown_connections 2024-09-21T01:20:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 -- 172.21.15.39:0/2345711201 wait complete. 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 -- start start 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.813+0000 7f5fe9a30740 1 Processor -- start 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0fe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d2e840 0x55eeda15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0fde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:35784/0 (socket says 172.21.15.39:35784) 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1ad310 con 0x55eed9d2e840 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 -- 172.21.15.39:0/3309889453 learned_addr learned my addr 172.21.15.39:0/3309889453 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1ae780 con 0x55eed9d05980 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1ad9e0 con 0x55eed9d0db60 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 10 client.0 _ll_register_callbacks cb 0x7f5fc8024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe0e00640 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d2e840 0x55eeda15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 -- 172.21.15.39:0/3309889453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 msgr2=0x55eeda0fe430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0fe430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 -- 172.21.15.39:0/3309889453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d2e840 msgr2=0x55eeda15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d2e840 0x55eeda15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 -- 172.21.15.39:0/3309889453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeda1602a0 con 0x55eed9d05980 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fdbe00640 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0fde70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc029510 tx=0x7f5fcc029540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a874ff02940a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fc6000640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fc6000640 1 -- 172.21.15.39:0/3309889453 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc022ae0 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fcc02e070 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fc6000640 1 -- 172.21.15.39:0/3309889453 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f5fcc032450 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fc6000640 1 -- 172.21.15.39:0/3309889453 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f5fcc032650 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 10 client.26204 fetch_fsmap learned FSMap version 51 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55eeda1675e0 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.819+0000 7f5fc6000640 1 -- 172.21.15.39:0/3309889453 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fcc02e070 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fc6000640 1 -- 172.21.15.39:0/3309889453 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 51) ==== 8370+0+0 (secure 0 0 0) 0x7f5fcc013070 con 0x55eed9d05980 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fe9a30740 10 client.26204 fetch_fsmap finished waiting for FSMap version 51 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fe9a30740 10 client.26204 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:20:01.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fe9a30740 20 client.26204 populate_metadata read hostname 'smithi039' 2024-09-21T01:20:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9cc2570 0x55eeda169e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fe9a30740 4 client.26204 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.2, "await_for": 45.0}] 2024-09-21T01:20:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.824+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.2, "await_for": 45.0}) -- 0x55eeda0496a0 con 0x55eed9cc2570 2024-09-21T01:20:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.825+0000 7f5fdb400640 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9cc2570 0x55eeda169e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:01.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.825+0000 7f5fdb400640 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9cc2570 0x55eeda169e70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2039a2d8cefe806f server_cookie=a34827f49f505eda in_seq=0 out_seq=0 2024-09-21T01:20:01.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:01.825+0000 7f5fc6000640 10 client.26204 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:01.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:01 smithi039 ceph-mon[30415]: pgmap v904: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 8.9 MiB/s wr, 217 op/s 2024-09-21T01:20:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:01 smithi150 ceph-mon[34945]: pgmap v904: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 8.9 MiB/s wr, 217 op/s 2024-09-21T01:20:02.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:01 smithi160 ceph-mon[34696]: pgmap v904: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 8.9 MiB/s wr, 217 op/s 2024-09-21T01:20:02.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 15.09 MB/sec execute 29 sec latency 19235.992 ms 2024-09-21T01:20:03.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 14.59 MB/sec execute 30 sec latency 20236.061 ms 2024-09-21T01:20:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:03 smithi150 ceph-mon[34945]: pgmap v905: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 7.2 MiB/s wr, 184 op/s 2024-09-21T01:20:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:03 smithi160 ceph-mon[34696]: pgmap v905: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 7.2 MiB/s wr, 184 op/s 2024-09-21T01:20:04.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:03 smithi039 ceph-mon[30415]: pgmap v905: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 7.2 MiB/s wr, 184 op/s 2024-09-21T01:20:04.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44035 14.12 MB/sec execute 31 sec latency 21236.126 ms 2024-09-21T01:20:05.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 13.67 MB/sec execute 32 sec latency 22236.183 ms 2024-09-21T01:20:05.998 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:05 smithi039 ceph-mon[30415]: pgmap v906: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 4.2 MiB/s wr, 127 op/s 2024-09-21T01:20:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:05 smithi160 ceph-mon[34696]: pgmap v906: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 4.2 MiB/s wr, 127 op/s 2024-09-21T01:20:06.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:05 smithi150 ceph-mon[34945]: pgmap v906: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 4.2 MiB/s wr, 127 op/s 2024-09-21T01:20:06.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 13.26 MB/sec execute 33 sec latency 23236.255 ms 2024-09-21T01:20:07.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 12.87 MB/sec execute 34 sec latency 24236.318 ms 2024-09-21T01:20:07.943 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:07 smithi039 ceph-mon[30415]: pgmap v907: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 343 KiB/s wr, 30 op/s 2024-09-21T01:20:08.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:07 smithi160 ceph-mon[34696]: pgmap v907: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 343 KiB/s wr, 30 op/s 2024-09-21T01:20:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:07 smithi150 ceph-mon[34945]: pgmap v907: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 343 KiB/s wr, 30 op/s 2024-09-21T01:20:08.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 12.50 MB/sec execute 35 sec latency 25236.382 ms 2024-09-21T01:20:09.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 12.16 MB/sec execute 36 sec latency 26236.446 ms 2024-09-21T01:20:10.041 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:09 smithi039 ceph-mon[30415]: pgmap v908: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 232 KiB/s wr, 55 op/s 2024-09-21T01:20:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:09 smithi160 ceph-mon[34696]: pgmap v908: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 232 KiB/s wr, 55 op/s 2024-09-21T01:20:10.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:09 smithi150 ceph-mon[34945]: pgmap v908: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 232 KiB/s wr, 55 op/s 2024-09-21T01:20:10.763 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 11.83 MB/sec execute 37 sec latency 27236.514 ms 2024-09-21T01:20:11.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 11.52 MB/sec execute 38 sec latency 28236.572 ms 2024-09-21T01:20:12.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:11 smithi039 ceph-mon[30415]: pgmap v909: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 136 KiB/s wr, 51 op/s 2024-09-21T01:20:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:11 smithi160 ceph-mon[34696]: pgmap v909: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 136 KiB/s wr, 51 op/s 2024-09-21T01:20:12.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:11 smithi150 ceph-mon[34945]: pgmap v909: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 136 KiB/s wr, 51 op/s 2024-09-21T01:20:12.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 11.22 MB/sec execute 39 sec latency 29236.636 ms 2024-09-21T01:20:13.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 10.94 MB/sec execute 40 sec latency 29276.873 ms 2024-09-21T01:20:14.050 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:13 smithi039 ceph-mon[30415]: pgmap v910: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 134 KiB/s wr, 50 op/s 2024-09-21T01:20:14.051 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:13 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:13 smithi160 ceph-mon[34696]: pgmap v910: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 134 KiB/s wr, 50 op/s 2024-09-21T01:20:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:13 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:13 smithi150 ceph-mon[34945]: pgmap v910: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 134 KiB/s wr, 50 op/s 2024-09-21T01:20:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:13 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:14.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 10.67 MB/sec execute 41 sec latency 29368.717 ms 2024-09-21T01:20:15.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44036 10.42 MB/sec execute 42 sec latency 30368.766 ms 2024-09-21T01:20:16.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:15 smithi039 ceph-mon[30415]: pgmap v911: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 147 KiB/s wr, 51 op/s 2024-09-21T01:20:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:15 smithi160 ceph-mon[34696]: pgmap v911: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 147 KiB/s wr, 51 op/s 2024-09-21T01:20:16.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:15 smithi150 ceph-mon[34945]: pgmap v911: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 147 KiB/s wr, 51 op/s 2024-09-21T01:20:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44037 10.18 MB/sec execute 43 sec latency 31368.813 ms 2024-09-21T01:20:17.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44037 9.95 MB/sec execute 44 sec latency 32368.876 ms 2024-09-21T01:20:18.023 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:17 smithi039 ceph-mon[30415]: pgmap v912: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 132 KiB/s wr, 49 op/s 2024-09-21T01:20:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:17 smithi160 ceph-mon[34696]: pgmap v912: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 132 KiB/s wr, 49 op/s 2024-09-21T01:20:18.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:17 smithi150 ceph-mon[34945]: pgmap v912: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 132 KiB/s wr, 49 op/s 2024-09-21T01:20:18.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44037 9.73 MB/sec execute 45 sec latency 33368.942 ms 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fc6000640 1 -- 172.21.15.39:0/3309889453 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55eeda0496a0 con 0x55eed9cc2570 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fc6000640 10 client.26204 handle_command_reply: tid=0 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 client.26204 shutdown 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 msgr2=0x55eeda0fde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0fde70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc029510 tx=0x7f5fcc029540 comp rx=0 tx=0).stop 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 shutdown_connections 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eed9cc2570 0x55eeda169e70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eed9d2e840 0x55eeda15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eed9d0db60 0x55eeda0fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 --2- 172.21.15.39:0/3309889453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed9d05980 0x55eeda0fde70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fc6000640 0 client.26204 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 >> 172.21.15.39:0/3309889453 conn(0x55eed9d22990 msgr2=0x55eeda0d74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 shutdown_connections 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.398+0000 7f5fe9a30740 1 -- 172.21.15.39:0/3309889453 wait complete. 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe9a30740 20 client.26204 trim_cache size 0 max 16384 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 52, 2024-09-21T01:20:19.401 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 62, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "7899c391": { 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 62, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 143.2, 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:20:19.402 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:19.403 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5fc00b6670 msgr2=0x7f5fc00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5fc00b6670 0x7f5fc00b8b80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55eeda0d5190 tx=0x7f5fcc008040 comp rx=0 tx=0).stop 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 msgr2=0x7f5fdc102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc102a70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800e9f0 tx=0x7f5fc800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 shutdown_connections 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fdc10dc50 0x7f5fdc102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5fc00b6670 0x7f5fc00b8b80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc105140 0x7f5fdc102a70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 --2- 172.21.15.39:0/3160984995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5fdc1047f0 0x7f5fdc102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.399+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 >> 172.21.15.39:0/3160984995 conn(0x7f5fdc0fe330 msgr2=0x7f5fdc1076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.400+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 shutdown_connections 2024-09-21T01:20:19.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:19.400+0000 7f5fe1800640 1 -- 172.21.15.39:0/3160984995 wait complete. 2024-09-21T01:20:19.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44038 9.51 MB/sec execute 46 sec latency 33786.782 ms 2024-09-21T01:20:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:19 smithi160 ceph-mon[34696]: pgmap v913: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 450 KiB/s wr, 75 op/s 2024-09-21T01:20:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:19 smithi039 ceph-mon[30415]: pgmap v913: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 450 KiB/s wr, 75 op/s 2024-09-21T01:20:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:19 smithi150 ceph-mon[34945]: pgmap v913: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 450 KiB/s wr, 75 op/s 2024-09-21T01:20:20.337 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7899c391', quiesce took 23.6 seconds. Will release after: -0.40000000000000213 2024-09-21T01:20:20.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:20:20.720 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:20:20.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44038 9.31 MB/sec execute 47 sec latency 29789.470 ms 2024-09-21T01:20:21.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 -- 172.21.15.39:0/1538755220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0078290 msgr2=0x7f78c0114200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:21.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 --2- 172.21.15.39:0/1538755220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0078290 0x7f78c0114200 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009900 tx=0x7f78a802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 -- 172.21.15.39:0/1538755220 shutdown_connections 2024-09-21T01:20:21.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 --2- 172.21.15.39:0/1538755220 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f78c0114740 0x7f78c0116b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 --2- 172.21.15.39:0/1538755220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0078290 0x7f78c0114200 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 --2- 172.21.15.39:0/1538755220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c0077940 0x7f78c0077d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 -- 172.21.15.39:0/1538755220 >> 172.21.15.39:0/1538755220 conn(0x7f78c0101810 msgr2=0x7f78c0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:21.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 -- 172.21.15.39:0/1538755220 shutdown_connections 2024-09-21T01:20:21.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.462+0000 7f78c5000640 1 -- 172.21.15.39:0/1538755220 wait complete. 2024-09-21T01:20:21.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.463+0000 7f78c5000640 1 Processor -- start 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.463+0000 7f78c5000640 1 -- start start 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78c5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 0x7f78c01a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78c5000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c0078290 0x7f78c01a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78c5000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f78c0114740 0x7f78c01a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78c5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f78c011a090 con 0x7f78c0077940 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78c5000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0119f10 con 0x7f78c0078290 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78c5000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f78c011a210 con 0x7f78c0114740 2024-09-21T01:20:21.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78bde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 0x7f78c01a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78be800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f78c0114740 0x7f78c01a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78bde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 0x7f78c01a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37544/0 (socket says 172.21.15.39:37544) 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78bde00640 1 -- 172.21.15.39:0/1117922585 learned_addr learned my addr 172.21.15.39:0/1117922585 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.464+0000 7f78b5400640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c0078290 0x7f78c01a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78bde00640 1 -- 172.21.15.39:0/1117922585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f78c0114740 msgr2=0x7f78c01a6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78bde00640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f78c0114740 0x7f78c01a6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78bde00640 1 -- 172.21.15.39:0/1117922585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c0078290 msgr2=0x7f78c01a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78bde00640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c0078290 0x7f78c01a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78bde00640 1 -- 172.21.15.39:0/1117922585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c01a6a90 con 0x7f78c0077940 2024-09-21T01:20:21.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78bde00640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 0x7f78c01a0b20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f78b000e9f0 tx=0x7f78b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b000cd80 con 0x7f78c0077940 2024-09-21T01:20:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b00108e0 con 0x7f78c0077940 2024-09-21T01:20:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.465+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c01a6d80 con 0x7f78c0077940 2024-09-21T01:20:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.466+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0018980 con 0x7f78c0077940 2024-09-21T01:20:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.466+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c01ad4e0 con 0x7f78c0077940 2024-09-21T01:20:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.467+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c0078e50 con 0x7f78c0077940 2024-09-21T01:20:21.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.469+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f78b0010430 con 0x7f78c0077940 2024-09-21T01:20:21.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.470+0000 7f78b7400640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78a40b6680 0x7f78a40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:21.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.471+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f78b00e8470 con 0x7f78c0077940 2024-09-21T01:20:21.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.471+0000 7f78b5400640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78a40b6680 0x7f78a40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:21.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.471+0000 7f78b5400640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78a40b6680 0x7f78a40b8b90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009230 tx=0x7f78a8031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:21.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.472+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f78b0026020 con 0x7f78c0077940 2024-09-21T01:20:21.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.708+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78c0077d50 con 0x7f78c0077940 2024-09-21T01:20:21.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.709+0000 7f78b7400640 1 -- 172.21.15.39:0/1117922585 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11915 (secure 0 0 0) 0x7f78b00b29c0 con 0x7f78c0077940 2024-09-21T01:20:21.710 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:20:21.710 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":52,"btime":"2024-09-21T01:20:12:833870+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:20:12.034241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:20:21.711 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 52 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.711+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78a40b6680 msgr2=0x7f78a40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.711+0000 7f78c5000640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78a40b6680 0x7f78a40b8b90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009230 tx=0x7f78a8031400 comp rx=0 tx=0).stop 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.711+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 msgr2=0x7f78c01a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.711+0000 7f78c5000640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 0x7f78c01a0b20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f78b000e9f0 tx=0x7f78b000eeb0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 shutdown_connections 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f78c0114740 0x7f78c01a6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f78a40b6680 0x7f78a40b8b90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c0078290 0x7f78c01a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 --2- 172.21.15.39:0/1117922585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78c0077940 0x7f78c01a0b20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 >> 172.21.15.39:0/1117922585 conn(0x7f78c0101810 msgr2=0x7f78c01167d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 shutdown_connections 2024-09-21T01:20:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:21.712+0000 7f78c5000640 1 -- 172.21.15.39:0/1117922585 wait complete. 2024-09-21T01:20:21.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44038 9.12 MB/sec execute 48 sec latency 30789.512 ms 2024-09-21T01:20:22.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:21 smithi039 ceph-mon[30415]: pgmap v914: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 383 KiB/s wr, 47 op/s 2024-09-21T01:20:22.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:21 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1117922585' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:20:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:21 smithi150 ceph-mon[34945]: pgmap v914: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 383 KiB/s wr, 47 op/s 2024-09-21T01:20:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:21 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1117922585' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:20:22.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:21 smithi160 ceph-mon[34696]: pgmap v914: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 383 KiB/s wr, 47 op/s 2024-09-21T01:20:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:21 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1117922585' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:20:22.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44038 8.93 MB/sec execute 49 sec latency 31789.587 ms 2024-09-21T01:20:23.011 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 7899c391 --release --await' 2024-09-21T01:20:23.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 7899c391 --release --await 2024-09-21T01:20:23.410 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:20:23.764 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44038 8.75 MB/sec execute 50 sec latency 32789.709 ms 2024-09-21T01:20:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: pgmap v915: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 435 KiB/s wr, 66 op/s 2024-09-21T01:20:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:23 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 -- 172.21.15.39:0/4092492564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 msgr2=0x7f607c075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 --2- 172.21.15.39:0/4092492564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c075b40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f606c009900 tx=0x7f606c02a410 comp rx=0 tx=0).stop 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 -- 172.21.15.39:0/4092492564 shutdown_connections 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 --2- 172.21.15.39:0/4092492564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f607c10ff90 0x7f607c1123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 --2- 172.21.15.39:0/4092492564 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f607c076080 0x7f607c10fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 --2- 172.21.15.39:0/4092492564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c075b40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 -- 172.21.15.39:0/4092492564 >> 172.21.15.39:0/4092492564 conn(0x7f607c0ff400 msgr2=0x7f607c101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 -- 172.21.15.39:0/4092492564 shutdown_connections 2024-09-21T01:20:24.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.226+0000 7f607be00640 1 -- 172.21.15.39:0/4092492564 wait complete. 2024-09-21T01:20:24.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.227+0000 7f607be00640 1 Processor -- start 2024-09-21T01:20:24.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 -- start start 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f607c076080 0x7f607c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f607c10ff90 0x7f607c1a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f607c115b30 con 0x7f607c075730 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1159b0 con 0x7f607c076080 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.228+0000 7f607be00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f607c115cb0 con 0x7f607c10ff90 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39236/0 (socket says 172.21.15.39:39236) 2024-09-21T01:20:24.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 -- 172.21.15.39:0/2293336041 learned_addr learned my addr 172.21.15.39:0/2293336041 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:24.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 -- 172.21.15.39:0/2293336041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f607c10ff90 msgr2=0x7f607c1a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f607c10ff90 0x7f607c1a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 -- 172.21.15.39:0/2293336041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f607c076080 msgr2=0x7f607c19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f607c076080 0x7f607c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.229+0000 7f607aa00640 1 -- 172.21.15.39:0/2293336041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c1a2470 con 0x7f607c075730 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.230+0000 7f607aa00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c19c500 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f606c005940 tx=0x7f606c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.230+0000 7f6073400640 1 -- 172.21.15.39:0/2293336041 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c03f030 con 0x7f607c075730 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.230+0000 7f6073400640 1 -- 172.21.15.39:0/2293336041 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f606c003cb0 con 0x7f607c075730 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.230+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c1a2700 con 0x7f607c075730 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.230+0000 7f6073400640 1 -- 172.21.15.39:0/2293336041 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c035070 con 0x7f607c075730 2024-09-21T01:20:24.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.230+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c1a8ec0 con 0x7f607c075730 2024-09-21T01:20:24.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 Processor -- start 2024-09-21T01:20:24.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 -- start start 2024-09-21T01:20:24.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6073400640 1 -- 172.21.15.39:0/2293336041 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f606c042030 con 0x7f607c075730 2024-09-21T01:20:24.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b30b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af3b660 0x55981b30b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b30bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55981b282290 con 0x55981af64520 2024-09-21T01:20:24.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55981b2716c0 con 0x55981af3b660 2024-09-21T01:20:24.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.234+0000 7f6088689740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55981b2754d0 con 0x55981af43840 2024-09-21T01:20:24.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b30b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b30b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39248/0 (socket says 172.21.15.39:39248) 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 -- 172.21.15.39:0/2775873334 learned_addr learned my addr 172.21.15.39:0/2775873334 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607aa00640 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af3b660 0x55981b30b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 -- 172.21.15.39:0/2775873334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 msgr2=0x55981b30bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b30bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 -- 172.21.15.39:0/2775873334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af3b660 msgr2=0x55981b30b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af3b660 0x55981b30b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.235+0000 7f607b400640 1 -- 172.21.15.39:0/2775873334 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981b1fe460 con 0x55981af64520 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.236+0000 7f607b400640 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b30b340 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6074009e50 tx=0x7f607400d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3d4fe8f59fbe1b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.236+0000 7f6067e00640 1 -- 172.21.15.39:0/2775873334 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074003b70 con 0x55981af64520 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.236+0000 7f6067e00640 1 -- 172.21.15.39:0/2775873334 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607400de20 con 0x55981af64520 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.236+0000 7f6067e00640 1 -- 172.21.15.39:0/2775873334 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074010820 con 0x55981af64520 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.236+0000 7f6073400640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f60600b6680 0x7f60600b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.236+0000 7f6073400640 1 -- 172.21.15.39:0/2293336041 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f606c1080c0 con 0x7f607c075730 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f607aa00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f60600b6680 0x7f60600b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 -- 172.21.15.39:0/2775873334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 msgr2=0x55981b30b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b30b340 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6074009e50 tx=0x7f607400d6b0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 -- 172.21.15.39:0/2775873334 shutdown_connections 2024-09-21T01:20:24.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b30bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af3b660 0x55981b30b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 --2- 172.21.15.39:0/2775873334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b30b340 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 -- 172.21.15.39:0/2775873334 >> 172.21.15.39:0/2775873334 conn(0x55981af58910 msgr2=0x55981b309230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:24.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.237+0000 7f6088689740 1 -- 172.21.15.39:0/2775873334 shutdown_connections 2024-09-21T01:20:24.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.238+0000 7f607aa00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f60600b6680 0x7f60600b8b90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f606c1054e0 tx=0x7f606c10bcf0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:24.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.238+0000 7f6088689740 1 -- 172.21.15.39:0/2775873334 wait complete. 2024-09-21T01:20:24.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.238+0000 7f6088689740 1 -- start start 2024-09-21T01:20:24.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.238+0000 7f6088689740 1 Processor -- start 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b316e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af43840 0x55981b3173c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b322e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55981b0d2bb0 con 0x55981af43840 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55981b20c6c0 con 0x55981af64520 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55981b319b20 con 0x55981af3b660 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f6088689740 10 client.0 _ll_register_callbacks cb 0x55981b227490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f607a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af43840 0x55981b3173c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f607a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af43840 0x55981b3173c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39262/0 (socket says 172.21.15.39:39262) 2024-09-21T01:20:24.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f607a000640 1 -- 172.21.15.39:0/3479855156 learned_addr learned my addr 172.21.15.39:0/3479855156 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:24.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.239+0000 7f607b400640 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b322e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f607b400640 1 -- 172.21.15.39:0/3479855156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 msgr2=0x55981b316e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:24.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f607b400640 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b316e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f607b400640 1 -- 172.21.15.39:0/3479855156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af43840 msgr2=0x55981b3173c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:24.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f607b400640 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af43840 0x55981b3173c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f607b400640 1 -- 172.21.15.39:0/3479855156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981afcfb00 con 0x55981af64520 2024-09-21T01:20:24.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f607b400640 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b322e10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6074005fd0 tx=0x7f6074009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=554681be3ac75aa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:24.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f6064c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:20:24.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.240+0000 7f6064c00640 1 -- 172.21.15.39:0/3479855156 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074004d20 con 0x55981af64520 2024-09-21T01:20:24.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.241+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f607400de20 con 0x55981af64520 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.241+0000 7f6064c00640 1 -- 172.21.15.39:0/3479855156 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f6074003970 con 0x55981af64520 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.241+0000 7f6064c00640 1 -- 172.21.15.39:0/3479855156 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f6074004470 con 0x55981af64520 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.241+0000 7f6088689740 10 client.26219 fetch_fsmap learned FSMap version 52 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.241+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55981b046960 con 0x55981af64520 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.241+0000 7f6064c00640 1 -- 172.21.15.39:0/3479855156 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607400de20 con 0x55981af64520 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6064c00640 1 -- 172.21.15.39:0/3479855156 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 52) ==== 8366+0+0 (secure 0 0 0) 0x7f6074023480 con 0x55981af64520 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6088689740 10 client.26219 fetch_fsmap finished waiting for FSMap version 52 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6088689740 10 client.26219 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6088689740 20 client.26219 populate_metadata read hostname 'smithi039' 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6088689740 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981af22ea0 0x55981b328cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6088689740 4 client.26219 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55981b2841e0 con 0x55981af22ea0 2024-09-21T01:20:24.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.242+0000 7f607a000640 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981af22ea0 0x55981b328cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.243+0000 7f607a000640 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981af22ea0 0x55981b328cb0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17f592b897112d0f server_cookie=6108c3438875a7ca in_seq=0 out_seq=0 2024-09-21T01:20:24.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.243+0000 7f6064c00640 10 client.26219 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.243+0000 7f6064c00640 1 -- 172.21.15.39:0/3479855156 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55981b2841e0 con 0x55981af22ea0 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.243+0000 7f6064c00640 10 client.26219 handle_command_reply: tid=0 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.243+0000 7f6088689740 1 client.26219 shutdown 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 msgr2=0x55981b322e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b322e10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6074005fd0 tx=0x7f6074009b90 comp rx=0 tx=0).stop 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 shutdown_connections 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981af22ea0 0x55981b328cb0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b322e10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af43840 0x55981b3173c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 --2- 172.21.15.39:0/3479855156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b316e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 >> 172.21.15.39:0/3479855156 conn(0x55981af58910 msgr2=0x55981b30e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6064c00640 0 client.26219 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 shutdown_connections 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 1 -- 172.21.15.39:0/3479855156 wait complete. 2024-09-21T01:20:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.244+0000 7f6088689740 20 client.26219 trim_cache size 0 max 16384 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.262+0000 7f6088689740 1 Processor -- start 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.262+0000 7f6088689740 1 -- start start 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f6088689740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 0x55981b322d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f6088689740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b323250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f6088689740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b310420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f6088689740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1b5900 con 0x55981af3b660 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f6088689740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1b31a0 con 0x55981af64520 2024-09-21T01:20:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f6088689740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1b1010 con 0x55981af43840 2024-09-21T01:20:24.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f607b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 0x55981b322d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f607b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 0x55981b322d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39270/0 (socket says 172.21.15.39:39270) 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f607b400640 1 -- 172.21.15.39:0/63928599 learned_addr learned my addr 172.21.15.39:0/63928599 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.263+0000 7f607a000640 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b323250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f607b400640 1 -- 172.21.15.39:0/63928599 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 msgr2=0x55981b323250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f607b400640 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b323250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f607b400640 1 -- 172.21.15.39:0/63928599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 msgr2=0x55981b310420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f607b400640 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b310420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f607b400640 1 -- 172.21.15.39:0/63928599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981b308960 con 0x55981af3b660 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f607b400640 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 0x55981b322d10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f60740048b0 tx=0x7f607400db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c248d0eb6e638c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f6067400640 1 -- 172.21.15.39:0/63928599 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074036070 con 0x55981af3b660 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f6067400640 1 -- 172.21.15.39:0/63928599 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6074004470 con 0x55981af3b660 2024-09-21T01:20:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.264+0000 7f6067400640 1 -- 172.21.15.39:0/63928599 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074023480 con 0x55981af3b660 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.265+0000 7f6088689740 1 -- 172.21.15.39:0/63928599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 msgr2=0x55981b322d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.265+0000 7f6088689740 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 0x55981b322d10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f60740048b0 tx=0x7f607400db30 comp rx=0 tx=0).stop 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 -- 172.21.15.39:0/63928599 shutdown_connections 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af64520 0x55981b310420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af43840 0x55981b323250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 --2- 172.21.15.39:0/63928599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af3b660 0x55981b322d10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 -- 172.21.15.39:0/63928599 >> 172.21.15.39:0/63928599 conn(0x55981af58910 msgr2=0x55981b30eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 -- 172.21.15.39:0/63928599 shutdown_connections 2024-09-21T01:20:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 -- 172.21.15.39:0/63928599 wait complete. 2024-09-21T01:20:24.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.266+0000 7f6088689740 1 -- start start 2024-09-21T01:20:24.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 Processor -- start 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b31ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 0x55981b31f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b319d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1b1660 con 0x55981af64520 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1b2ad0 con 0x55981af43840 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1b1d30 con 0x55981af3b660 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.267+0000 7f6088689740 10 client.0 _ll_register_callbacks cb 0x7f606c10bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 0x55981b31f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 0x55981b31f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:59106/0 (socket says 172.21.15.39:59106) 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 -- 172.21.15.39:0/2338220692 learned_addr learned my addr 172.21.15.39:0/2338220692 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607b400640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b319d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607b400640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b31ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 -- 172.21.15.39:0/2338220692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 msgr2=0x55981b31ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b31ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 -- 172.21.15.39:0/2338220692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 msgr2=0x55981b319d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b319d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 -- 172.21.15.39:0/2338220692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981b31a370 con 0x55981af43840 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f607a000640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 0x55981b31f4c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55981b329410 tx=0x7f6068009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=970332c349e91d75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.268+0000 7f6064c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.269+0000 7f6064c00640 1 -- 172.21.15.39:0/2338220692 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068021070 con 0x55981af43840 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.269+0000 7f6064c00640 1 -- 172.21.15.39:0/2338220692 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9511+0+0 (secure 0 0 0) 0x7f6068016cf0 con 0x55981af43840 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.269+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6074004470 con 0x55981af43840 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.269+0000 7f6064c00640 1 -- 172.21.15.39:0/2338220692 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f606802b050 con 0x55981af43840 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.270+0000 7f6088689740 10 client.26222 fetch_fsmap learned FSMap version 52 2024-09-21T01:20:24.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.270+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55981b393fd0 con 0x55981af43840 2024-09-21T01:20:24.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.270+0000 7f6064c00640 1 -- 172.21.15.39:0/2338220692 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6074004470 con 0x55981af43840 2024-09-21T01:20:24.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.270+0000 7f6064c00640 1 -- 172.21.15.39:0/2338220692 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 52) ==== 8366+0+0 (secure 0 0 0) 0x7f606801b040 con 0x55981af43840 2024-09-21T01:20:24.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f6088689740 10 client.26222 fetch_fsmap finished waiting for FSMap version 52 2024-09-21T01:20:24.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f6088689740 10 client.26222 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:20:24.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f6088689740 20 client.26222 populate_metadata read hostname 'smithi039' 2024-09-21T01:20:24.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f6088689740 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981aef8250 0x55981b396850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:20:24.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f6088689740 4 client.26222 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7899c391", "release": true, "await": true}] 2024-09-21T01:20:24.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7899c391", "release": true, "await": true}) -- 0x55981b27fc90 con 0x55981aef8250 2024-09-21T01:20:24.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.271+0000 7f607b400640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981aef8250 0x55981b396850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:20:24.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.272+0000 7f607b400640 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981aef8250 0x55981b396850 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd5b524bf7857d64 server_cookie=f6228bb9d3806060 in_seq=0 out_seq=0 2024-09-21T01:20:24.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.273+0000 7f6064c00640 10 client.26222 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: pgmap v915: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 435 KiB/s wr, 66 op/s 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:23 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.316+0000 7f6064c00640 1 -- 172.21.15.39:0/2338220692 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55981b27fc90 con 0x55981aef8250 2024-09-21T01:20:24.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.316+0000 7f6064c00640 10 client.26222 handle_command_reply: tid=0 2024-09-21T01:20:24.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.316+0000 7f6088689740 1 client.26222 shutdown 2024-09-21T01:20:24.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 msgr2=0x55981b31f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 0x55981b31f4c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55981b329410 tx=0x7f6068009ec0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 shutdown_connections 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55981aef8250 0x55981b396850 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55981af64520 0x55981b319d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55981af43840 0x55981b31f4c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 --2- 172.21.15.39:0/2338220692 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55981af3b660 0x55981b31ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 >> 172.21.15.39:0/2338220692 conn(0x55981af58910 msgr2=0x55981b30eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6064c00640 0 client.26222 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 shutdown_connections 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.317+0000 7f6088689740 1 -- 172.21.15.39:0/2338220692 wait complete. 2024-09-21T01:20:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.318+0000 7f6088689740 20 client.26222 trim_cache size 0 max 16384 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 52, 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 64, 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "7899c391": { 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 64, 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:20:24.320 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 143.2, 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:20:24.321 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:20:24.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.319+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f60600b6680 msgr2=0x7f60600b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.319+0000 7f607be00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f60600b6680 0x7f60600b8b90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f606c1054e0 tx=0x7f606c10bcf0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.319+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 msgr2=0x7f607c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:20:24.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.319+0000 7f607be00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c19c500 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f606c005940 tx=0x7f606c0042f0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.320+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 shutdown_connections 2024-09-21T01:20:24.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.320+0000 7f607be00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f607c10ff90 0x7f607c1a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.320+0000 7f607be00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f60600b6680 0x7f60600b8b90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.320+0000 7f607be00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f607c076080 0x7f607c19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.320+0000 7f607be00640 1 --2- 172.21.15.39:0/2293336041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f607c075730 0x7f607c19c500 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:20:24.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.320+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 >> 172.21.15.39:0/2293336041 conn(0x7f607c0ff400 msgr2=0x7f607c101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:20:24.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.322+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 shutdown_connections 2024-09-21T01:20:24.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:20:24.322+0000 7f607be00640 1 -- 172.21.15.39:0/2293336041 wait complete. 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: pgmap v915: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 435 KiB/s wr, 66 op/s 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:23 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T01:20:24.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 44363 8.98 MB/sec execute 51 sec latency 33318.494 ms 2024-09-21T01:20:25.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi160 to 176.0M 2024-09-21T01:20:25.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi150 to 176.0M 2024-09-21T01:20:25.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi160 to 184592486: error parsing value: Value '184592486' is below minimum 939524096 2024-09-21T01:20:25.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi150 to 184591769: error parsing value: Value '184591769' is below minimum 939524096 2024-09-21T01:20:25.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:20:25.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:20:25.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:20:25.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi160 to 176.0M 2024-09-21T01:20:25.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi150 to 176.0M 2024-09-21T01:20:25.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi160 to 184592486: error parsing value: Value '184592486' is below minimum 939524096 2024-09-21T01:20:25.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi150 to 184591769: error parsing value: Value '184591769' is below minimum 939524096 2024-09-21T01:20:25.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:20:25.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:20:25.315 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi160 to 176.0M 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi150 to 176.0M 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi160 to 184592486: error parsing value: Value '184592486' is below minimum 939524096 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi150 to 184591769: error parsing value: Value '184591769' is below minimum 939524096 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:20:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:20:25.458 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7899c391', total seconds elapsed: 28.8 2024-09-21T01:20:25.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 45065 9.48 MB/sec execute 52 sec latency 69.014 ms 2024-09-21T01:20:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:25 smithi039 ceph-mon[30415]: pgmap v916: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 485 KiB/s wr, 107 op/s 2024-09-21T01:20:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:25 smithi039 ceph-mon[30415]: osdmap e111: 12 total, 12 up, 12 in 2024-09-21T01:20:26.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:25 smithi150 ceph-mon[34945]: pgmap v916: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 485 KiB/s wr, 107 op/s 2024-09-21T01:20:26.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:25 smithi150 ceph-mon[34945]: osdmap e111: 12 total, 12 up, 12 in 2024-09-21T01:20:26.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:25 smithi160 ceph-mon[34696]: pgmap v916: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 485 KiB/s wr, 107 op/s 2024-09-21T01:20:26.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:25 smithi160 ceph-mon[34696]: osdmap e111: 12 total, 12 up, 12 in 2024-09-21T01:20:26.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 45811 10.03 MB/sec execute 53 sec latency 51.994 ms 2024-09-21T01:20:27.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 46628 10.57 MB/sec execute 54 sec latency 54.850 ms 2024-09-21T01:20:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:27 smithi039 ceph-mon[30415]: pgmap v918: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 684 KiB/s wr, 133 op/s 2024-09-21T01:20:28.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:27 smithi150 ceph-mon[34945]: pgmap v918: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 684 KiB/s wr, 133 op/s 2024-09-21T01:20:28.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:27 smithi160 ceph-mon[34696]: pgmap v918: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 684 KiB/s wr, 133 op/s 2024-09-21T01:20:28.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 47498 11.44 MB/sec execute 55 sec latency 75.564 ms 2024-09-21T01:20:29.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 48210 11.96 MB/sec execute 56 sec latency 151.429 ms 2024-09-21T01:20:30.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:29 smithi039 ceph-mon[30415]: pgmap v919: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.3 MiB/s wr, 252 op/s 2024-09-21T01:20:30.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:29 smithi150 ceph-mon[34945]: pgmap v919: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.3 MiB/s wr, 252 op/s 2024-09-21T01:20:30.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:29 smithi160 ceph-mon[34696]: pgmap v919: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.3 MiB/s wr, 252 op/s 2024-09-21T01:20:30.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 48872 12.29 MB/sec execute 57 sec latency 153.730 ms 2024-09-21T01:20:31.765 INFO:tasks.workunit.client.0.smithi039.stdout: 10 49811 12.73 MB/sec execute 58 sec latency 44.733 ms 2024-09-21T01:20:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:31 smithi039 ceph-mon[30415]: pgmap v920: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 324 op/s 2024-09-21T01:20:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:31 smithi150 ceph-mon[34945]: pgmap v920: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 324 op/s 2024-09-21T01:20:32.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:31 smithi160 ceph-mon[34696]: pgmap v920: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 324 op/s 2024-09-21T01:20:32.766 INFO:tasks.workunit.client.0.smithi039.stdout: 10 50753 13.68 MB/sec execute 59 sec latency 80.870 ms 2024-09-21T01:20:33.766 INFO:tasks.workunit.client.0.smithi039.stdout: 10 51461 13.93 MB/sec execute 60 sec latency 87.305 ms 2024-09-21T01:20:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:33 smithi039 ceph-mon[30415]: pgmap v921: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-09-21T01:20:34.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:33 smithi150 ceph-mon[34945]: pgmap v921: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-09-21T01:20:34.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:33 smithi160 ceph-mon[34696]: pgmap v921: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-09-21T01:20:34.766 INFO:tasks.workunit.client.0.smithi039.stdout: 10 52131 14.30 MB/sec execute 61 sec latency 94.855 ms 2024-09-21T01:20:35.766 INFO:tasks.workunit.client.0.smithi039.stdout: 10 53211 14.86 MB/sec execute 62 sec latency 55.630 ms 2024-09-21T01:20:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:35 smithi150 ceph-mon[34945]: pgmap v922: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 419 op/s 2024-09-21T01:20:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:35 smithi150 ceph-mon[34945]: osdmap e112: 12 total, 12 up, 12 in 2024-09-21T01:20:35.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:35 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:35.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:35 smithi160 ceph-mon[34696]: pgmap v922: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 419 op/s 2024-09-21T01:20:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:35 smithi160 ceph-mon[34696]: osdmap e112: 12 total, 12 up, 12 in 2024-09-21T01:20:35.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:35 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:35 smithi039 ceph-mon[30415]: pgmap v922: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 419 op/s 2024-09-21T01:20:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:35 smithi039 ceph-mon[30415]: osdmap e112: 12 total, 12 up, 12 in 2024-09-21T01:20:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:35 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:36.768 INFO:tasks.workunit.client.0.smithi039.stdout: 10 54182 15.77 MB/sec execute 63 sec latency 46.860 ms 2024-09-21T01:20:37.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 54697 15.82 MB/sec execute 64 sec latency 81.013 ms 2024-09-21T01:20:37.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:37 smithi160 ceph-mon[34696]: pgmap v924: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.6 MiB/s wr, 513 op/s 2024-09-21T01:20:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:37 smithi039 ceph-mon[30415]: pgmap v924: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.6 MiB/s wr, 513 op/s 2024-09-21T01:20:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:37 smithi150 ceph-mon[34945]: pgmap v924: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.6 MiB/s wr, 513 op/s 2024-09-21T01:20:38.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 55461 16.26 MB/sec execute 65 sec latency 116.782 ms 2024-09-21T01:20:39.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:39 smithi039 ceph-mon[30415]: pgmap v925: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.8 MiB/s wr, 545 op/s 2024-09-21T01:20:39.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 56292 16.55 MB/sec execute 66 sec latency 238.596 ms 2024-09-21T01:20:39.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:39 smithi160 ceph-mon[34696]: pgmap v925: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.8 MiB/s wr, 545 op/s 2024-09-21T01:20:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:39 smithi150 ceph-mon[34945]: pgmap v925: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.8 MiB/s wr, 545 op/s 2024-09-21T01:20:40.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 56864 16.84 MB/sec execute 67 sec latency 424.910 ms 2024-09-21T01:20:41.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 57566 17.21 MB/sec execute 68 sec latency 455.362 ms 2024-09-21T01:20:41.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:41 smithi160 ceph-mon[34696]: pgmap v926: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.1 MiB/s wr, 541 op/s 2024-09-21T01:20:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:41 smithi039 ceph-mon[30415]: pgmap v926: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.1 MiB/s wr, 541 op/s 2024-09-21T01:20:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:41 smithi150 ceph-mon[34945]: pgmap v926: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.1 MiB/s wr, 541 op/s 2024-09-21T01:20:42.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 58283 17.42 MB/sec execute 69 sec latency 106.966 ms 2024-09-21T01:20:43.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 59140 17.76 MB/sec execute 70 sec latency 137.608 ms 2024-09-21T01:20:43.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:43 smithi160 ceph-mon[34696]: pgmap v927: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 526 op/s 2024-09-21T01:20:43.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:43 smithi039 ceph-mon[30415]: pgmap v927: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 526 op/s 2024-09-21T01:20:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:43 smithi150 ceph-mon[34945]: pgmap v927: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 526 op/s 2024-09-21T01:20:44.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 60300 18.56 MB/sec execute 71 sec latency 25.639 ms 2024-09-21T01:20:45.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 60429 18.44 MB/sec execute 72 sec latency 891.125 ms 2024-09-21T01:20:45.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:45 smithi039 ceph-mon[30415]: pgmap v928: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 584 op/s 2024-09-21T01:20:45.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:45 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:45.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:45 smithi160 ceph-mon[34696]: pgmap v928: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 584 op/s 2024-09-21T01:20:45.857 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:45 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:45 smithi150 ceph-mon[34945]: pgmap v928: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 584 op/s 2024-09-21T01:20:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:45 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:20:46.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 60474 18.21 MB/sec execute 73 sec latency 1891.188 ms 2024-09-21T01:20:47.748 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:47 smithi039 ceph-mon[30415]: pgmap v929: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 508 op/s 2024-09-21T01:20:47.767 INFO:tasks.workunit.client.0.smithi039.stdout: 10 60554 18.05 MB/sec execute 74 sec latency 2701.149 ms 2024-09-21T01:20:47.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:47 smithi160 ceph-mon[34696]: pgmap v929: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 508 op/s 2024-09-21T01:20:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:47 smithi150 ceph-mon[34945]: pgmap v929: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 508 op/s 2024-09-21T01:20:48.768 INFO:tasks.workunit.client.0.smithi039.stdout: 10 60593 17.81 MB/sec execute 75 sec latency 3699.146 ms 2024-09-21T01:20:49.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:49 smithi039 ceph-mon[30415]: pgmap v930: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 572 op/s 2024-09-21T01:20:49.768 INFO:tasks.workunit.client.0.smithi039.stdout: 10 60950 17.79 MB/sec execute 76 sec latency 4345.897 ms 2024-09-21T01:20:49.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:49 smithi160 ceph-mon[34696]: pgmap v930: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 572 op/s 2024-09-21T01:20:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:49 smithi150 ceph-mon[34945]: pgmap v930: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 572 op/s 2024-09-21T01:20:50.768 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61054 17.69 MB/sec execute 77 sec latency 784.509 ms 2024-09-21T01:20:51.768 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61060 17.46 MB/sec execute 78 sec latency 1666.201 ms 2024-09-21T01:20:51.922 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:51 smithi039 ceph-mon[30415]: pgmap v931: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.2 MiB/s wr, 446 op/s 2024-09-21T01:20:52.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:51 smithi150 ceph-mon[34945]: pgmap v931: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.2 MiB/s wr, 446 op/s 2024-09-21T01:20:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:51 smithi160 ceph-mon[34696]: pgmap v931: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.2 MiB/s wr, 446 op/s 2024-09-21T01:20:52.768 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61064 17.24 MB/sec execute 79 sec latency 1262.665 ms 2024-09-21T01:20:53.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61065 17.03 MB/sec execute 80 sec latency 2187.144 ms 2024-09-21T01:20:53.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:53 smithi039 ceph-mon[30415]: pgmap v932: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 11 MiB/s wr, 449 op/s 2024-09-21T01:20:54.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:53 smithi150 ceph-mon[34945]: pgmap v932: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 11 MiB/s wr, 449 op/s 2024-09-21T01:20:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:53 smithi160 ceph-mon[34696]: pgmap v932: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 11 MiB/s wr, 449 op/s 2024-09-21T01:20:54.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61067 16.82 MB/sec execute 81 sec latency 3187.705 ms 2024-09-21T01:20:55.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61096 16.62 MB/sec execute 82 sec latency 4048.420 ms 2024-09-21T01:20:55.800 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:55 smithi039 ceph-mon[30415]: pgmap v933: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 15 MiB/s wr, 540 op/s 2024-09-21T01:20:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:55 smithi150 ceph-mon[34945]: pgmap v933: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 15 MiB/s wr, 540 op/s 2024-09-21T01:20:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:55 smithi160 ceph-mon[34696]: pgmap v933: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 15 MiB/s wr, 540 op/s 2024-09-21T01:20:56.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 16.49 MB/sec execute 83 sec latency 4909.653 ms 2024-09-21T01:20:57.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 16.30 MB/sec execute 84 sec latency 5438.930 ms 2024-09-21T01:20:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:57 smithi150 ceph-mon[34945]: pgmap v934: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 15 MiB/s wr, 454 op/s 2024-09-21T01:20:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:57 smithi160 ceph-mon[34696]: pgmap v934: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 15 MiB/s wr, 454 op/s 2024-09-21T01:20:58.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:57 smithi039 ceph-mon[30415]: pgmap v934: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 15 MiB/s wr, 454 op/s 2024-09-21T01:20:58.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 16.10 MB/sec execute 85 sec latency 6438.975 ms 2024-09-21T01:20:59.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 15.92 MB/sec execute 86 sec latency 7439.020 ms 2024-09-21T01:21:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:20:59 smithi150 ceph-mon[34945]: pgmap v935: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 15 MiB/s wr, 546 op/s 2024-09-21T01:21:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:20:59 smithi160 ceph-mon[34696]: pgmap v935: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 15 MiB/s wr, 546 op/s 2024-09-21T01:21:00.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:20:59 smithi039 ceph-mon[30415]: pgmap v935: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 15 MiB/s wr, 546 op/s 2024-09-21T01:21:00.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 15.73 MB/sec execute 87 sec latency 8439.082 ms 2024-09-21T01:21:01.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 15.55 MB/sec execute 88 sec latency 9439.130 ms 2024-09-21T01:21:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:01 smithi150 ceph-mon[34945]: pgmap v936: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 415 op/s 2024-09-21T01:21:02.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:01 smithi160 ceph-mon[34696]: pgmap v936: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 415 op/s 2024-09-21T01:21:02.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:01 smithi039 ceph-mon[30415]: pgmap v936: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 415 op/s 2024-09-21T01:21:02.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 15.38 MB/sec execute 89 sec latency 10439.173 ms 2024-09-21T01:21:03.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 15.21 MB/sec execute 90 sec latency 11439.227 ms 2024-09-21T01:21:03.955 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:03 smithi039 ceph-mon[30415]: pgmap v937: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 11 MiB/s wr, 393 op/s 2024-09-21T01:21:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:03 smithi150 ceph-mon[34945]: pgmap v937: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 11 MiB/s wr, 393 op/s 2024-09-21T01:21:04.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:03 smithi160 ceph-mon[34696]: pgmap v937: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 11 MiB/s wr, 393 op/s 2024-09-21T01:21:04.769 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 15.04 MB/sec execute 91 sec latency 12439.282 ms 2024-09-21T01:21:05.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 14.88 MB/sec execute 92 sec latency 13439.338 ms 2024-09-21T01:21:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:05 smithi150 ceph-mon[34945]: pgmap v938: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 340 op/s 2024-09-21T01:21:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:05 smithi150 ceph-mon[34945]: osdmap e113: 12 total, 12 up, 12 in 2024-09-21T01:21:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:05 smithi160 ceph-mon[34696]: pgmap v938: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 340 op/s 2024-09-21T01:21:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:05 smithi160 ceph-mon[34696]: osdmap e113: 12 total, 12 up, 12 in 2024-09-21T01:21:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:05 smithi039 ceph-mon[30415]: pgmap v938: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 340 op/s 2024-09-21T01:21:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:05 smithi039 ceph-mon[30415]: osdmap e113: 12 total, 12 up, 12 in 2024-09-21T01:21:06.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 14.72 MB/sec execute 93 sec latency 14439.384 ms 2024-09-21T01:21:07.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 14.56 MB/sec execute 94 sec latency 15439.430 ms 2024-09-21T01:21:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:07 smithi150 ceph-mon[34945]: pgmap v940: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 568 KiB/s wr, 187 op/s 2024-09-21T01:21:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:07 smithi160 ceph-mon[34696]: pgmap v940: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 568 KiB/s wr, 187 op/s 2024-09-21T01:21:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:07 smithi039 ceph-mon[30415]: pgmap v940: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 568 KiB/s wr, 187 op/s 2024-09-21T01:21:08.545 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-09-21T01:21:08.545 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:21:08.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 14.41 MB/sec execute 95 sec latency 16439.478 ms 2024-09-21T01:21:08.939 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:21:09.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 14.26 MB/sec execute 96 sec latency 17439.524 ms 2024-09-21T01:21:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:09 smithi150 ceph-mon[34945]: pgmap v941: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 339 KiB/s wr, 43 op/s 2024-09-21T01:21:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:09 smithi160 ceph-mon[34696]: pgmap v941: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 339 KiB/s wr, 43 op/s 2024-09-21T01:21:10.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:09 smithi039 ceph-mon[30415]: pgmap v941: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 339 KiB/s wr, 43 op/s 2024-09-21T01:21:10.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61176 14.11 MB/sec execute 97 sec latency 18439.576 ms 2024-09-21T01:21:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:10 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:21:11.074 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:10 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:21:11.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:10 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:21:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 -- 172.21.15.39:0/1218867802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e8103b60 msgr2=0x7ff6e8103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 --2- 172.21.15.39:0/1218867802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e8103b60 0x7ff6e8103f70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009900 tx=0x7ff6dc02a410 comp rx=0 tx=0).stop 2024-09-21T01:21:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 -- 172.21.15.39:0/1218867802 shutdown_connections 2024-09-21T01:21:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 --2- 172.21.15.39:0/1218867802 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8109df0 0x7ff6e8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 --2- 172.21.15.39:0/1218867802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e81094a0 0x7ff6e81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 --2- 172.21.15.39:0/1218867802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e8103b60 0x7ff6e8103f70 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 -- 172.21.15.39:0/1218867802 >> 172.21.15.39:0/1218867802 conn(0x7ff6e807b850 msgr2=0x7ff6e807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 -- 172.21.15.39:0/1218867802 shutdown_connections 2024-09-21T01:21:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.544+0000 7ff6f0400640 1 -- 172.21.15.39:0/1218867802 wait complete. 2024-09-21T01:21:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.545+0000 7ff6f0400640 1 Processor -- start 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 -- start start 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 0x7ff6e819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e81094a0 0x7ff6e819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e8109df0 0x7ff6e81a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8113ab0 con 0x7ff6e81094a0 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8113930 con 0x7ff6e8109df0 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6f0400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8113c30 con 0x7ff6e8103b60 2024-09-21T01:21:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6eda00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 0x7ff6e819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6eda00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 0x7ff6e819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:54782/0 (socket says 172.21.15.39:54782) 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6eda00640 1 -- 172.21.15.39:0/3098634363 learned_addr learned my addr 172.21.15.39:0/3098634363 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.546+0000 7ff6ed000640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e81094a0 0x7ff6e819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6ee400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e8109df0 0x7ff6e81a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6eda00640 1 -- 172.21.15.39:0/3098634363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e8109df0 msgr2=0x7ff6e81a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6eda00640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e8109df0 0x7ff6e81a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6eda00640 1 -- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e81094a0 msgr2=0x7ff6e819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6eda00640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e81094a0 0x7ff6e819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6eda00640 1 -- 172.21.15.39:0/3098634363 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e81a25f0 con 0x7ff6e8103b60 2024-09-21T01:21:11.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.547+0000 7ff6ee400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e8109df0 0x7ff6e81a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:21:11.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.551+0000 7ff6eda00640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 0x7ff6e819c680 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc005940 tx=0x7ff6dc0040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:11.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.551+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc03f030 con 0x7ff6e8103b60 2024-09-21T01:21:11.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.551+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6dc0044c0 con 0x7ff6e8103b60 2024-09-21T01:21:11.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.551+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc035070 con 0x7ff6e8103b60 2024-09-21T01:21:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.551+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e81a2880 con 0x7ff6e8103b60 2024-09-21T01:21:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.552+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e81a9040 con 0x7ff6e8103b60 2024-09-21T01:21:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.553+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e8103f70 con 0x7ff6e8103b60 2024-09-21T01:21:11.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.557+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff6dc042030 con 0x7ff6e8103b60 2024-09-21T01:21:11.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.559+0000 7ff6e6000640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6d00b6700 0x7ff6d00b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:11.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.559+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9599+0+0 (secure 0 0 0) 0x7ff6dc108140 con 0x7ff6e8103b60 2024-09-21T01:21:11.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.560+0000 7ff6ed000640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6d00b6700 0x7ff6d00b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:11.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.560+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff6dc0c9dd0 con 0x7ff6e8103b60 2024-09-21T01:21:11.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.560+0000 7ff6ed000640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6d00b6700 0x7ff6d00b8c10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40095f0 tx=0x7ff6d4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:11.770 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61344 14.03 MB/sec execute 98 sec latency 19382.004 ms 2024-09-21T01:21:11.867 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:11 smithi039 ceph-mon[30415]: pgmap v942: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 88 KiB/s wr, 31 op/s 2024-09-21T01:21:11.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.938+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6e819d7b0 con 0x7ff6e8103b60 2024-09-21T01:21:11.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.941+0000 7ff6e6000640 1 -- 172.21.15.39:0/3098634363 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11911 (secure 0 0 0) 0x7ff6dc0ce260 con 0x7ff6e8103b60 2024-09-21T01:21:11.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:21:11.943 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 55 2024-09-21T01:21:11.944 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":55,"btime":"2024-09-21T01:21:09:751771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:09.385142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6d00b6700 msgr2=0x7ff6d00b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6d00b6700 0x7ff6d00b8c10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40095f0 tx=0x7ff6d4005f90 comp rx=0 tx=0).stop 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 msgr2=0x7ff6e819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 0x7ff6e819c680 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc005940 tx=0x7ff6dc0040c0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 shutdown_connections 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff6d00b6700 0x7ff6d00b8c10 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40095f0 tx=0x7ff6d4005f90 comp rx=0 tx=0).stop 2024-09-21T01:21:11.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6e8109df0 0x7ff6e81a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6e81094a0 0x7ff6e819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 --2- 172.21.15.39:0/3098634363 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff6e8103b60 0x7ff6e819c680 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:11.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.945+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 >> 172.21.15.39:0/3098634363 conn(0x7ff6e807b850 msgr2=0x7ff6e81073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:11.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.947+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 shutdown_connections 2024-09-21T01:21:11.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:11.947+0000 7ff6f0400640 1 -- 172.21.15.39:0/3098634363 wait complete. 2024-09-21T01:21:12.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:11 smithi150 ceph-mon[34945]: pgmap v942: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 88 KiB/s wr, 31 op/s 2024-09-21T01:21:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:11 smithi160 ceph-mon[34696]: pgmap v942: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 88 KiB/s wr, 31 op/s 2024-09-21T01:21:12.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61776 13.99 MB/sec execute 99 sec latency 325.953 ms 2024-09-21T01:21:13.040 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-09-21T01:21:13.040 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-09-21T01:21:13.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:12 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3098634363' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:13.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:12 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3098634363' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:13.129 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:12 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3098634363' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:21:13.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61876 13.89 MB/sec execute 100 sec latency 555.323 ms 2024-09-21T01:21:13.799 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:13 smithi039 ceph-mon[30415]: pgmap v943: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 239 KiB/s wr, 33 op/s 2024-09-21T01:21:14.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:13 smithi150 ceph-mon[34945]: pgmap v943: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 239 KiB/s wr, 33 op/s 2024-09-21T01:21:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:13 smithi160 ceph-mon[34696]: pgmap v943: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 239 KiB/s wr, 33 op/s 2024-09-21T01:21:14.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 61993 13.89 MB/sec execute 101 sec latency 327.564 ms 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.825+0000 7feb08600640 1 -- 172.21.15.39:0/642782194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb001094c0 msgr2=0x7feb001098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.825+0000 7feb08600640 1 --2- 172.21.15.39:0/642782194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb001094c0 0x7feb001098d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7feaec009860 tx=0x7feaec02a340 comp rx=0 tx=0).stop 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.827+0000 7feb08600640 1 -- 172.21.15.39:0/642782194 shutdown_connections 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.827+0000 7feb08600640 1 --2- 172.21.15.39:0/642782194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb00109e10 0x7feb00110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.827+0000 7feb08600640 1 --2- 172.21.15.39:0/642782194 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb001094c0 0x7feb001098d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.827+0000 7feb08600640 1 --2- 172.21.15.39:0/642782194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 0x7feb00103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.827+0000 7feb08600640 1 -- 172.21.15.39:0/642782194 >> 172.21.15.39:0/642782194 conn(0x7feb000ff5d0 msgr2=0x7feb00101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:14.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.829+0000 7feb08600640 1 -- 172.21.15.39:0/642782194 shutdown_connections 2024-09-21T01:21:14.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.829+0000 7feb08600640 1 -- 172.21.15.39:0/642782194 wait complete. 2024-09-21T01:21:14.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.831+0000 7feb08600640 1 Processor -- start 2024-09-21T01:21:14.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.831+0000 7feb08600640 1 -- start start 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.832+0000 7feb08600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 0x7feb0019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb08600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb001094c0 0x7feb0019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb08600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 0x7feb001a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb08600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7feb00113af0 con 0x7feb00103b80 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb08600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb00113970 con 0x7feb001094c0 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb08600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feb00113c70 con 0x7feb00109e10 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb06600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 0x7feb001a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb06600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 0x7feb001a1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36728/0 (socket says 172.21.15.39:36728) 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.833+0000 7feb06600640 1 -- 172.21.15.39:0/2239814788 learned_addr learned my addr 172.21.15.39:0/2239814788 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:14.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb05200640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb001094c0 0x7feb0019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb05c00640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 0x7feb0019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb06600640 1 -- 172.21.15.39:0/2239814788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb001094c0 msgr2=0x7feb0019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:14.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb06600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb001094c0 0x7feb0019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb06600640 1 -- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 msgr2=0x7feb0019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:14.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb06600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 0x7feb0019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb06600640 1 -- 172.21.15.39:0/2239814788 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb001a26f0 con 0x7feb00109e10 2024-09-21T01:21:14.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.834+0000 7feb05c00640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 0x7feb0019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:21:14.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.835+0000 7feb05200640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb001094c0 0x7feb0019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:21:14.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.835+0000 7feb06600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 0x7feb001a1fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7feaf800b970 tx=0x7feaf800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:14.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.835+0000 7feafe000640 1 -- 172.21.15.39:0/2239814788 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8019070 con 0x7feb00109e10 2024-09-21T01:21:14.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.835+0000 7feafe000640 1 -- 172.21.15.39:0/2239814788 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf8014440 con 0x7feb00109e10 2024-09-21T01:21:14.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.835+0000 7feafe000640 1 -- 172.21.15.39:0/2239814788 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf80075d0 con 0x7feb00109e10 2024-09-21T01:21:14.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.838+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb001a29e0 con 0x7feb00109e10 2024-09-21T01:21:14.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.838+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb001a9140 con 0x7feb00109e10 2024-09-21T01:21:14.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.843+0000 7feb0f3c8740 1 Processor -- start 2024-09-21T01:21:14.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.848+0000 7feafe000640 1 -- 172.21.15.39:0/2239814788 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7feaf801c070 con 0x7feb00109e10 2024-09-21T01:21:14.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.848+0000 7feb0f3c8740 1 -- start start 2024-09-21T01:21:14.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.849+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb30b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.849+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb310a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.849+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb315e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.849+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55749ea9dca0 con 0x55749e75fb60 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.849+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55749ea8d0d0 con 0x55749e757980 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.849+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55749ea90ee0 con 0x55749e780840 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb30b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb30b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36734/0 (socket says 172.21.15.39:36734) 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 -- 172.21.15.39:0/673942012 learned_addr learned my addr 172.21.15.39:0/673942012 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05c00640 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 -- 172.21.15.39:0/673942012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 msgr2=0x55749eb310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 -- 172.21.15.39:0/673942012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 msgr2=0x55749eb315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb315e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.850+0000 7feb05200640 1 -- 172.21.15.39:0/673942012 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55749ea3f400 con 0x55749e780840 2024-09-21T01:21:14.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.851+0000 7feb05200640 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb30b60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb0006baa0 tx=0x7feaec009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded5f91ae85647de server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:14.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.851+0000 7feaf2a00640 1 -- 172.21.15.39:0/673942012 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec037040 con 0x55749e780840 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.851+0000 7feaf2a00640 1 -- 172.21.15.39:0/673942012 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaec065070 con 0x55749e780840 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.852+0000 7feaf2a00640 1 -- 172.21.15.39:0/673942012 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec037040 con 0x55749e780840 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.852+0000 7feb0f3c8740 1 -- 172.21.15.39:0/673942012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 msgr2=0x55749eb30b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.852+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb30b60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb0006baa0 tx=0x7feaec009490 comp rx=0 tx=0).stop 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 -- 172.21.15.39:0/673942012 shutdown_connections 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/673942012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb30b60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 -- 172.21.15.39:0/673942012 >> 172.21.15.39:0/673942012 conn(0x55749e774990 msgr2=0x55749eb2ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 -- 172.21.15.39:0/673942012 shutdown_connections 2024-09-21T01:21:14.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.853+0000 7feb0f3c8740 1 -- 172.21.15.39:0/673942012 wait complete. 2024-09-21T01:21:14.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.858+0000 7feafe000640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feae80b6690 0x7feae80b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.858+0000 7feafe000640 1 -- 172.21.15.39:0/2239814788 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7feaf80e70b0 con 0x7feb00109e10 2024-09-21T01:21:14.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.859+0000 7feb0f3c8740 1 -- start start 2024-09-21T01:21:14.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.860+0000 7feb05c00640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feae80b6690 0x7feae80b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.860+0000 7feb0f3c8740 1 Processor -- start 2024-09-21T01:21:14.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb3c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e75fb60 0x55749eb3c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb3cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55749e8ee7a0 con 0x55749e757980 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55749ea32a10 con 0x55749e75fb60 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55749eb3f120 con 0x55749e780840 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb0f3c8740 10 client.0 _ll_register_callbacks cb 0x55749ea42be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb06600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb3cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb06600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb3cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36744/0 (socket says 172.21.15.39:36744) 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.861+0000 7feb06600640 1 -- 172.21.15.39:0/526998529 learned_addr learned my addr 172.21.15.39:0/526998529 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feb06600640 1 -- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e75fb60 msgr2=0x55749eb3c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feb06600640 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e75fb60 0x55749eb3c6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feb06600640 1 -- 172.21.15.39:0/526998529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 msgr2=0x55749eb3c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feb06600640 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb3c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feb06600640 1 -- 172.21.15.39:0/526998529 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55749e7fab90 con 0x55749e780840 2024-09-21T01:21:14.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feb06600640 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb3cc60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feaf80a9090 tx=0x7feaf80ac230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86903e6f35a72ea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.862+0000 7feae7400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feb05c00640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feae80b6690 0x7feae80b8ba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55749eb2f7e0 tx=0x7feaf4009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feae7400640 1 -- 172.21.15.39:0/526998529 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf80e4840 con 0x55749e780840 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feae7400640 1 -- 172.21.15.39:0/526998529 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7feaf800cb40 con 0x55749e780840 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaec065070 con 0x55749e780840 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feae7400640 1 -- 172.21.15.39:0/526998529 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7feaf80ad1c0 con 0x55749e780840 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feb0f3c8740 10 client.26224 fetch_fsmap learned FSMap version 55 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.863+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55749eb4ae40 con 0x55749e780840 2024-09-21T01:21:14.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feae7400640 1 -- 172.21.15.39:0/526998529 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaec065070 con 0x55749e780840 2024-09-21T01:21:14.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feae7400640 1 -- 172.21.15.39:0/526998529 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7feaf80b11a0 con 0x55749e780840 2024-09-21T01:21:14.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feb0f3c8740 10 client.26224 fetch_fsmap finished waiting for FSMap version 55 2024-09-21T01:21:14.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feb0f3c8740 10 client.26224 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feb0f3c8740 20 client.26224 populate_metadata read hostname 'smithi039' 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e73f1c0 0x55749eb4e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feb0f3c8740 4 client.26224 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.864+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55749ea9fbf0 con 0x55749e73f1c0 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.865+0000 7feb05c00640 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e73f1c0 0x55749eb4e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.866+0000 7feae7400640 10 client.26224 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:14.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.866+0000 7feb05c00640 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e73f1c0 0x55749eb4e310 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8b89e365e69e74 server_cookie=903a501207d6bab0 in_seq=0 out_seq=0 2024-09-21T01:21:14.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.867+0000 7feae7400640 1 -- 172.21.15.39:0/526998529 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55749ea9fbf0 con 0x55749e73f1c0 2024-09-21T01:21:14.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.867+0000 7feae7400640 10 client.26224 handle_command_reply: tid=0 2024-09-21T01:21:14.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.867+0000 7feb0f3c8740 1 client.26224 shutdown 2024-09-21T01:21:14.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.868+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 msgr2=0x55749eb3cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:14.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.868+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb3cc60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feaf80a9090 tx=0x7feaf80ac230 comp rx=0 tx=0).stop 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 shutdown_connections 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e73f1c0 0x55749eb4e310 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb3cc60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e75fb60 0x55749eb3c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/526998529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb3c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feae7400640 0 client.26224 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:14.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 >> 172.21.15.39:0/526998529 conn(0x55749e774990 msgr2=0x55749eb33cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:14.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 shutdown_connections 2024-09-21T01:21:14.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.869+0000 7feb0f3c8740 1 -- 172.21.15.39:0/526998529 wait complete. 2024-09-21T01:21:14.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.870+0000 7feb0f3c8740 20 client.26224 trim_cache size 0 max 16384 2024-09-21T01:21:14.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 Processor -- start 2024-09-21T01:21:14.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 -- start start 2024-09-21T01:21:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb3c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb2d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb393b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7feb00074fe0 con 0x55749e75fb60 2024-09-21T01:21:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb000728b0 con 0x55749e757980 2024-09-21T01:21:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.909+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feb00070800 con 0x55749e780840 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb3c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb3c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:36896/0 (socket says 172.21.15.39:36896) 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 -- 172.21.15.39:0/1648669650 learned_addr learned my addr 172.21.15.39:0/1648669650 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05200640 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb2d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 -- 172.21.15.39:0/1648669650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 msgr2=0x55749eb393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 -- 172.21.15.39:0/1648669650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 msgr2=0x55749eb2d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb2d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.910+0000 7feb05c00640 1 -- 172.21.15.39:0/1648669650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55749eb21c50 con 0x55749e757980 2024-09-21T01:21:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.911+0000 7feb05c00640 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb3c880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feaf40096c0 tx=0x7feaf401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df3cf2efb7f27eda server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.911+0000 7feaf2000640 1 -- 172.21.15.39:0/1648669650 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4011e50 con 0x55749e757980 2024-09-21T01:21:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.911+0000 7feaf2000640 1 -- 172.21.15.39:0/1648669650 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf401b070 con 0x55749e757980 2024-09-21T01:21:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.911+0000 7feaf2000640 1 -- 172.21.15.39:0/1648669650 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4010730 con 0x55749e757980 2024-09-21T01:21:14.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1648669650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 msgr2=0x55749eb3c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:14.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb3c880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feaf40096c0 tx=0x7feaf401b000 comp rx=0 tx=0).stop 2024-09-21T01:21:14.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1648669650 shutdown_connections 2024-09-21T01:21:14.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e780840 0x55749eb393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e75fb60 0x55749eb2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1648669650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e757980 0x55749eb3c880 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.913+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1648669650 >> 172.21.15.39:0/1648669650 conn(0x55749e774990 msgr2=0x55749eb3c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:14.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.914+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1648669650 shutdown_connections 2024-09-21T01:21:14.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.914+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1648669650 wait complete. 2024-09-21T01:21:14.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.916+0000 7feb0f3c8740 1 -- start start 2024-09-21T01:21:14.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.916+0000 7feb0f3c8740 1 Processor -- start 2024-09-21T01:21:14.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.917+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb44ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e75fb60 0x55749eb411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb0f3c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e780840 0x55749eb41800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7feb00070e50 con 0x55749e757980 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb000721e0 con 0x55749e780840 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb0f3c8740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feb000714a0 con 0x55749e75fb60 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb05c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb44ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb05c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb44ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54386/0 (socket says 172.21.15.39:54386) 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.918+0000 7feb05c00640 1 -- 172.21.15.39:0/1701205214 learned_addr learned my addr 172.21.15.39:0/1701205214 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb0f3c8740 10 client.0 _ll_register_callbacks cb 0x7feaf8014c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb05200640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e75fb60 0x55749eb411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb06600640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e780840 0x55749eb41800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb05200640 1 -- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e780840 msgr2=0x55749eb41800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb05200640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e780840 0x55749eb41800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb05200640 1 -- 172.21.15.39:0/1701205214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 msgr2=0x55749eb44ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:14.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb05200640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb44ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:14.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.919+0000 7feb05200640 1 -- 172.21.15.39:0/1701205214 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55749eb41f70 con 0x55749e75fb60 2024-09-21T01:21:14.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.920+0000 7feb05200640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e75fb60 0x55749eb411b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feaec009780 tx=0x7feaec009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14da9e66e640d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:14.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.921+0000 7feae7400640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:21:14.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.921+0000 7feae7400640 1 -- 172.21.15.39:0/1701205214 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec037040 con 0x55749e75fb60 2024-09-21T01:21:14.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.921+0000 7feae7400640 1 -- 172.21.15.39:0/1701205214 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7feaec004450 con 0x55749e75fb60 2024-09-21T01:21:14.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.921+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaf401b070 con 0x55749e75fb60 2024-09-21T01:21:14.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.921+0000 7feae7400640 1 -- 172.21.15.39:0/1701205214 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7feaec030450 con 0x55749e75fb60 2024-09-21T01:21:14.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.922+0000 7feb0f3c8740 10 client.26227 fetch_fsmap learned FSMap version 55 2024-09-21T01:21:14.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.922+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55749ebb13c0 con 0x55749e75fb60 2024-09-21T01:21:14.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.923+0000 7feae7400640 1 -- 172.21.15.39:0/1701205214 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf401b070 con 0x55749e75fb60 2024-09-21T01:21:14.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.923+0000 7feae7400640 1 -- 172.21.15.39:0/1701205214 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7feaec03d040 con 0x55749e75fb60 2024-09-21T01:21:14.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.923+0000 7feb0f3c8740 10 client.26227 fetch_fsmap finished waiting for FSMap version 55 2024-09-21T01:21:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.924+0000 7feb0f3c8740 10 client.26227 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:21:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.924+0000 7feb0f3c8740 20 client.26227 populate_metadata read hostname 'smithi039' 2024-09-21T01:21:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.924+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e714570 0x55749ebb3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.924+0000 7feb0f3c8740 4 client.26227 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-09-21T01:21:14.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.924+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55749ea9b6a0 con 0x55749e714570 2024-09-21T01:21:14.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.925+0000 7feb05c00640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e714570 0x55749ebb3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:14.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.927+0000 7feb05c00640 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e714570 0x55749ebb3600 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddb913c4cca0f9de server_cookie=ab9586669d020a52 in_seq=0 out_seq=0 2024-09-21T01:21:14.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:14.927+0000 7feae7400640 10 client.26227 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feae7400640 1 -- 172.21.15.39:0/1701205214 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55749ea9b6a0 con 0x55749e714570 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feae7400640 10 client.26227 handle_command_reply: tid=0 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 client.26227 shutdown 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e75fb60 msgr2=0x55749eb411b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e75fb60 0x55749eb411b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feaec009780 tx=0x7feaec009520 comp rx=0 tx=0).stop 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 shutdown_connections 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749e714570 0x55749ebb3600 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749e780840 0x55749eb41800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feae7400640 0 client.26227 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749e75fb60 0x55749eb411b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 --2- 172.21.15.39:0/1701205214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749e757980 0x55749eb44ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.494+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 >> 172.21.15.39:0/1701205214 conn(0x55749e774990 msgr2=0x55749eb47360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.495+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 shutdown_connections 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.495+0000 7feb0f3c8740 1 -- 172.21.15.39:0/1701205214 wait complete. 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.495+0000 7feb0f3c8740 20 client.26227 trim_cache size 0 max 16384 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 55, 2024-09-21T01:21:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 66, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "c745c3b7": { 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 66, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 130.0, 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:21:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:15.499 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feae80b6690 msgr2=0x7feae80b8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feae80b6690 0x7feae80b8ba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55749eb2f7e0 tx=0x7feaf4009260 comp rx=0 tx=0).stop 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 msgr2=0x7feb001a1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 0x7feb001a1fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7feaf800b970 tx=0x7feaf800be30 comp rx=0 tx=0).stop 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 shutdown_connections 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feb00109e10 0x7feb001a1fc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7feae80b6690 0x7feae80b8ba0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb001094c0 0x7feb0019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 --2- 172.21.15.39:0/2239814788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb00103b80 0x7feb0019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 >> 172.21.15.39:0/2239814788 conn(0x7feb000ff5d0 msgr2=0x7feb00108b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 shutdown_connections 2024-09-21T01:21:15.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:15.496+0000 7feb08600640 1 -- 172.21.15.39:0/2239814788 wait complete. 2024-09-21T01:21:15.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.76 MB/sec execute 102 sec latency 713.042 ms 2024-09-21T01:21:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:15 smithi160 ceph-mon[34696]: pgmap v944: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 530 KiB/s wr, 44 op/s 2024-09-21T01:21:15.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:15 smithi160 ceph-mon[34696]: osdmap e114: 12 total, 12 up, 12 in 2024-09-21T01:21:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:15 smithi039 ceph-mon[30415]: pgmap v944: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 530 KiB/s wr, 44 op/s 2024-09-21T01:21:15.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:15 smithi039 ceph-mon[30415]: osdmap e114: 12 total, 12 up, 12 in 2024-09-21T01:21:16.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:15 smithi150 ceph-mon[34945]: pgmap v944: 129 pgs: 129 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 530 KiB/s wr, 44 op/s 2024-09-21T01:21:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:15 smithi150 ceph-mon[34945]: osdmap e114: 12 total, 12 up, 12 in 2024-09-21T01:21:16.624 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c745c3b7', quiesce took 8.1 seconds. Will release after: 1.9000000000000004 2024-09-21T01:21:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.63 MB/sec execute 103 sec latency 1713.164 ms 2024-09-21T01:21:17.772 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.50 MB/sec execute 104 sec latency 2713.309 ms 2024-09-21T01:21:17.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:17 smithi160 ceph-mon[34696]: pgmap v946: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-09-21T01:21:17.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:17 smithi039 ceph-mon[30415]: pgmap v946: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-09-21T01:21:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:17 smithi150 ceph-mon[34945]: pgmap v946: 129 pgs: 129 active+clean; 1.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-09-21T01:21:18.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:21:18.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.37 MB/sec execute 105 sec latency 3713.404 ms 2024-09-21T01:21:18.920 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:21:19.639 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:19 smithi039 ceph-mon[30415]: pgmap v947: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-09-21T01:21:19.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 -- 172.21.15.39:0/1587804746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f81039b0 msgr2=0x7fe9f8103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:19.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 --2- 172.21.15.39:0/1587804746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f81039b0 0x7fe9f8103dc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009900 tx=0x7fe9ec02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 -- 172.21.15.39:0/1587804746 shutdown_connections 2024-09-21T01:21:19.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 --2- 172.21.15.39:0/1587804746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9f8109c40 0x7fe9f8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 --2- 172.21.15.39:0/1587804746 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81092f0 0x7fe9f8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 --2- 172.21.15.39:0/1587804746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f81039b0 0x7fe9f8103dc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.712+0000 7fe9ff200640 1 -- 172.21.15.39:0/1587804746 >> 172.21.15.39:0/1587804746 conn(0x7fe9f80ff440 msgr2=0x7fe9f8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:19.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.713+0000 7fe9ff200640 1 -- 172.21.15.39:0/1587804746 shutdown_connections 2024-09-21T01:21:19.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.713+0000 7fe9ff200640 1 -- 172.21.15.39:0/1587804746 wait complete. 2024-09-21T01:21:19.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 Processor -- start 2024-09-21T01:21:19.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 -- start start 2024-09-21T01:21:19.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 0x7fe9f819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:19.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9f81092f0 0x7fe9f819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:19.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f8109c40 0x7fe9f819fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:19.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f8113900 con 0x7fe9f8109c40 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f8113780 con 0x7fe9f81092f0 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.714+0000 7fe9ff200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f8113a80 con 0x7fe9f81039b0 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 0x7fe9f819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 0x7fe9f819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:36762/0 (socket says 172.21.15.39:36762) 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 -- 172.21.15.39:0/2470682009 learned_addr learned my addr 172.21.15.39:0/2470682009 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fe800640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f8109c40 0x7fe9f819fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 -- 172.21.15.39:0/2470682009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9f81092f0 msgr2=0x7fe9f819a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9f81092f0 0x7fe9f819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 -- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f8109c40 msgr2=0x7fe9f819fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f8109c40 0x7fe9f819fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 -- 172.21.15.39:0/2470682009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f81a0260 con 0x7fe9f81039b0 2024-09-21T01:21:19.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.715+0000 7fe9fde00640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 0x7fe9f819a2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec005940 tx=0x7fe9ec004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:19.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.716+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec03f030 con 0x7fe9f81039b0 2024-09-21T01:21:19.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.716+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9ec003aa0 con 0x7fe9f81039b0 2024-09-21T01:21:19.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.716+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f81a04f0 con 0x7fe9f81039b0 2024-09-21T01:21:19.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.716+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec035070 con 0x7fe9f81039b0 2024-09-21T01:21:19.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.716+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f81a6cb0 con 0x7fe9f81039b0 2024-09-21T01:21:19.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.717+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f8103dc0 con 0x7fe9f81039b0 2024-09-21T01:21:19.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.719+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fe9ec042030 con 0x7fe9f81039b0 2024-09-21T01:21:19.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.721+0000 7fe9f2a00640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe9e00b6680 0x7fe9e00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:19.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.721+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fe9ec108560 con 0x7fe9f81039b0 2024-09-21T01:21:19.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.721+0000 7fe9fd400640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe9e00b6680 0x7fe9e00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:19.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.722+0000 7fe9fd400640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe9e00b6680 0x7fe9e00b8b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80058c0 tx=0x7fe9e8005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:19.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.723+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe9ec0ce670 con 0x7fe9f81039b0 2024-09-21T01:21:19.771 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.25 MB/sec execute 106 sec latency 4713.451 ms 2024-09-21T01:21:19.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:19 smithi160 ceph-mon[34696]: pgmap v947: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-09-21T01:21:19.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:19 smithi150 ceph-mon[34945]: pgmap v947: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-09-21T01:21:19.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.963+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9f819b420 con 0x7fe9f81039b0 2024-09-21T01:21:19.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.964+0000 7fe9f2a00640 1 -- 172.21.15.39:0/2470682009 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11911 (secure 0 0 0) 0x7fe9ec0d2a50 con 0x7fe9f81039b0 2024-09-21T01:21:19.965 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:21:19.965 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":55,"btime":"2024-09-21T01:21:09:751771+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:09.385142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:21:19.966 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 55 2024-09-21T01:21:19.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe9e00b6680 msgr2=0x7fe9e00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:19.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe9e00b6680 0x7fe9e00b8b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80058c0 tx=0x7fe9e8005850 comp rx=0 tx=0).stop 2024-09-21T01:21:19.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 msgr2=0x7fe9f819a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:19.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 0x7fe9f819a2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec005940 tx=0x7fe9ec004270 comp rx=0 tx=0).stop 2024-09-21T01:21:19.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 shutdown_connections 2024-09-21T01:21:19.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9f8109c40 0x7fe9f819fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe9e00b6680 0x7fe9e00b8b90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9f81092f0 0x7fe9f819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 --2- 172.21.15.39:0/2470682009 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe9f81039b0 0x7fe9f819a2f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:19.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 >> 172.21.15.39:0/2470682009 conn(0x7fe9f80ff440 msgr2=0x7fe9f8101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:19.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 shutdown_connections 2024-09-21T01:21:19.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:19.967+0000 7fe9ff200640 1 -- 172.21.15.39:0/2470682009 wait complete. 2024-09-21T01:21:20.772 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.12 MB/sec execute 107 sec latency 5713.556 ms 2024-09-21T01:21:20.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:20 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2470682009' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:20.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:20 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2470682009' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:21.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:20 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2470682009' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:21.458 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id c745c3b7 --release --await' 2024-09-21T01:21:21.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id c745c3b7 --release --await 2024-09-21T01:21:21.772 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 13.00 MB/sec execute 108 sec latency 6713.627 ms 2024-09-21T01:21:21.848 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:21:21.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:21 smithi160 ceph-mon[34696]: pgmap v948: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 299 op/s 2024-09-21T01:21:21.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:21 smithi039 ceph-mon[30415]: pgmap v948: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 299 op/s 2024-09-21T01:21:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:21 smithi150 ceph-mon[34945]: pgmap v948: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 299 op/s 2024-09-21T01:21:22.772 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62005 12.88 MB/sec execute 109 sec latency 7713.683 ms 2024-09-21T01:21:23.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 -- 172.21.15.39:0/3100305291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8103990 msgr2=0x7f39d8103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 --2- 172.21.15.39:0/3100305291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8103990 0x7f39d8103da0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009ca0 tx=0x7f39c802a8c0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 -- 172.21.15.39:0/3100305291 shutdown_connections 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 --2- 172.21.15.39:0/3100305291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39d8109c20 0x7f39d8110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 --2- 172.21.15.39:0/3100305291 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d81092d0 0x7f39d81096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 --2- 172.21.15.39:0/3100305291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8103990 0x7f39d8103da0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 -- 172.21.15.39:0/3100305291 >> 172.21.15.39:0/3100305291 conn(0x7f39d80ff440 msgr2=0x7f39d8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 -- 172.21.15.39:0/3100305291 shutdown_connections 2024-09-21T01:21:23.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.106+0000 7f39dd400640 1 -- 172.21.15.39:0/3100305291 wait complete. 2024-09-21T01:21:23.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.107+0000 7f39dd400640 1 Processor -- start 2024-09-21T01:21:23.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.107+0000 7f39dd400640 1 -- start start 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dd400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39d8103990 0x7f39d819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d81092d0 0x7f39d819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dd400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d8109c20 0x7f39d819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dd400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81138e0 con 0x7f39d81092d0 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dd400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39d8113760 con 0x7f39d8103990 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dd400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39d8113a60 con 0x7f39d8109c20 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d81092d0 0x7f39d819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d81092d0 0x7f39d819a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42220/0 (socket says 172.21.15.39:42220) 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39cf400640 1 -- 172.21.15.39:0/3916576305 learned_addr learned my addr 172.21.15.39:0/3916576305 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39dca00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d8109c20 0x7f39d819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.108+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39d8103990 0x7f39d819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dca00640 1 -- 172.21.15.39:0/3916576305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39d8103990 msgr2=0x7f39d819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dca00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39d8103990 0x7f39d819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dca00640 1 -- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d81092d0 msgr2=0x7f39d819a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dca00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d81092d0 0x7f39d819a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dca00640 1 -- 172.21.15.39:0/3916576305 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d81a0240 con 0x7f39d8109c20 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dca00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d8109c20 0x7f39d819fb10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f39d000e970 tx=0x7f39d000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39d5600640 1 -- 172.21.15.39:0/3916576305 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d000ccc0 con 0x7f39d8109c20 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d81a0530 con 0x7f39d8109c20 2024-09-21T01:21:23.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39d5600640 1 -- 172.21.15.39:0/3916576305 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39d0005020 con 0x7f39d8109c20 2024-09-21T01:21:23.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.109+0000 7f39d5600640 1 -- 172.21.15.39:0/3916576305 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0004a00 con 0x7f39d8109c20 2024-09-21T01:21:23.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.110+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d81a6c90 con 0x7f39d8109c20 2024-09-21T01:21:23.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.113+0000 7f39e55ef740 1 Processor -- start 2024-09-21T01:21:23.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.113+0000 7f39d5600640 1 -- 172.21.15.39:0/3916576305 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f39d0004ba0 con 0x7f39d8109c20 2024-09-21T01:21:23.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.113+0000 7f39e55ef740 1 -- start start 2024-09-21T01:21:23.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4c2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39e55ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55985d439290 con 0x55985d0f2660 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39e55ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55985d4286c0 con 0x55985d11b520 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39e55ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55985d42c4d0 con 0x55985d0fa840 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39cf400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39cf400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39544/0 (socket says 172.21.15.39:39544) 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39cf400640 1 -- 172.21.15.39:0/2642701772 learned_addr learned my addr 172.21.15.39:0/2642701772 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39d7e00640 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4c2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.114+0000 7f39cf400640 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.115+0000 7f39cf400640 1 -- 172.21.15.39:0/2642701772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 msgr2=0x55985d4c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.115+0000 7f39cf400640 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.115+0000 7f39cf400640 1 -- 172.21.15.39:0/2642701772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 msgr2=0x55985d4c2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.115+0000 7f39cf400640 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4c2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.115+0000 7f39cf400640 1 -- 172.21.15.39:0/2642701772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985d3b5460 con 0x55985d11b520 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.115+0000 7f39cf400640 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c2340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f39c00095e0 tx=0x7f39c0009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c54a7b80a4bdcf81 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.116+0000 7f39cd200640 1 -- 172.21.15.39:0/2642701772 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c000f7c0 con 0x55985d11b520 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.116+0000 7f39cd200640 1 -- 172.21.15.39:0/2642701772 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c000fe10 con 0x55985d11b520 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.116+0000 7f39cd200640 1 -- 172.21.15.39:0/2642701772 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c000ecc0 con 0x55985d11b520 2024-09-21T01:21:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.116+0000 7f39d5600640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39bc0b6700 0x7f39bc0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.116+0000 7f39d5600640 1 -- 172.21.15.39:0/3916576305 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7f39d00e7a70 con 0x7f39d8109c20 2024-09-21T01:21:23.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.116+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39bc0b6700 0x7f39bc0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 -- 172.21.15.39:0/2642701772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 msgr2=0x55985d4c2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c2340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f39c00095e0 tx=0x7f39c0009570 comp rx=0 tx=0).stop 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39bc0b6700 0x7f39bc0b8c10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55985d4c0fc0 tx=0x7f39c8009360 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 -- 172.21.15.39:0/2642701772 shutdown_connections 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4c2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2642701772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c2340 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 -- 172.21.15.39:0/2642701772 >> 172.21.15.39:0/2642701772 conn(0x55985d10f910 msgr2=0x55985d4c0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 -- 172.21.15.39:0/2642701772 shutdown_connections 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 -- 172.21.15.39:0/2642701772 wait complete. 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.117+0000 7f39e55ef740 1 -- start start 2024-09-21T01:21:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 Processor -- start 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55985d289bb0 con 0x55985d0f2660 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55985d3c36c0 con 0x55985d11b520 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55985d4d0b20 con 0x55985d0fa840 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39e55ef740 10 client.0 _ll_register_callbacks cb 0x55985d3de490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.118+0000 7f39cf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:35992/0 (socket says 172.21.15.39:35992) 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 -- 172.21.15.39:0/2880617432 learned_addr learned my addr 172.21.15.39:0/2880617432 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39dca00640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4d9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 -- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 msgr2=0x55985d4d9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4d9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 -- 172.21.15.39:0/2880617432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 msgr2=0x55985d4cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 -- 172.21.15.39:0/2880617432 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985d186b00 con 0x55985d0fa840 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39dca00640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39cf400640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4ce4e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f39c00099c0 tx=0x7f39c000e940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b40afecfaf7cfb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.119+0000 7f39c5600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39c5600640 1 -- 172.21.15.39:0/2880617432 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0025070 con 0x55985d0fa840 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39c5600640 1 -- 172.21.15.39:0/2880617432 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7f39c000ea40 con 0x55985d0fa840 2024-09-21T01:21:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39c000fe10 con 0x55985d0fa840 2024-09-21T01:21:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39c5600640 1 -- 172.21.15.39:0/2880617432 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f39c000ec40 con 0x55985d0fa840 2024-09-21T01:21:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39e55ef740 10 client.26245 fetch_fsmap learned FSMap version 55 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55985d1fd960 con 0x55985d0fa840 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.120+0000 7f39c5600640 1 -- 172.21.15.39:0/2880617432 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c000fe10 con 0x55985d0fa840 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39c5600640 1 -- 172.21.15.39:0/2880617432 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7f39c001d020 con 0x55985d0fa840 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39e55ef740 10 client.26245 fetch_fsmap finished waiting for FSMap version 55 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39e55ef740 10 client.26245 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39e55ef740 20 client.26245 populate_metadata read hostname 'smithi039' 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0d9ea0 0x55985d4dfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39e55ef740 4 client.26245 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55985d43b1e0 con 0x55985d0d9ea0 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.121+0000 7f39d7e00640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0d9ea0 0x55985d4dfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.122+0000 7f39d7e00640 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0d9ea0 0x55985d4dfcf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ab4af048dcc3e9f server_cookie=2fd65a90044e91fe in_seq=0 out_seq=0 2024-09-21T01:21:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.122+0000 7f39c5600640 10 client.26245 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39c5600640 1 -- 172.21.15.39:0/2880617432 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55985d43b1e0 con 0x55985d0d9ea0 2024-09-21T01:21:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39c5600640 10 client.26245 handle_command_reply: tid=0 2024-09-21T01:21:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 client.26245 shutdown 2024-09-21T01:21:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 msgr2=0x55985d4ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4ce4e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f39c00099c0 tx=0x7f39c000e940 comp rx=0 tx=0).stop 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 shutdown_connections 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0d9ea0 0x55985d4dfcf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4ce4e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 --2- 172.21.15.39:0/2880617432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0f2660 0x55985d4cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 >> 172.21.15.39:0/2880617432 conn(0x55985d10f910 msgr2=0x55985d4c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39c5600640 0 client.26245 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.123+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 shutdown_connections 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.124+0000 7f39e55ef740 1 -- 172.21.15.39:0/2880617432 wait complete. 2024-09-21T01:21:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.124+0000 7f39e55ef740 20 client.26245 trim_cache size 0 max 16384 2024-09-21T01:21:23.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 Processor -- start 2024-09-21T01:21:23.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 -- start start 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0f2660 0x55985d4d9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 0x55985d4bfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81b3660 con 0x55985d0fa840 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81b0f30 con 0x55985d11b520 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39e55ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81aeda0 con 0x55985d0f2660 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 0x55985d4bfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 0x55985d4bfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42254/0 (socket says 172.21.15.39:42254) 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.143+0000 7f39cf400640 1 -- 172.21.15.39:0/3254147852 learned_addr learned my addr 172.21.15.39:0/3254147852 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0f2660 0x55985d4d9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39dca00640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 -- 172.21.15.39:0/3254147852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 msgr2=0x55985d4c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 -- 172.21.15.39:0/3254147852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 msgr2=0x55985d4bfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 0x55985d4bfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 -- 172.21.15.39:0/3254147852 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985d4c5b20 con 0x55985d0f2660 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39cf400640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 0x55985d4bfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:21:23.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39dca00640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:21:23.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39d7e00640 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0f2660 0x55985d4d9d10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f39c8000f80 tx=0x7f39c8044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12c48215fb5f3143 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:23.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.144+0000 7f39c7e00640 1 -- 172.21.15.39:0/3254147852 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8031760 con 0x55985d0f2660 2024-09-21T01:21:23.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39c7e00640 1 -- 172.21.15.39:0/3254147852 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c8036050 con 0x55985d0f2660 2024-09-21T01:21:23.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39c7e00640 1 -- 172.21.15.39:0/3254147852 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8004780 con 0x55985d0f2660 2024-09-21T01:21:23.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39e55ef740 1 -- 172.21.15.39:0/3254147852 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0f2660 msgr2=0x55985d4d9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39e55ef740 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0f2660 0x55985d4d9d10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f39c8000f80 tx=0x7f39c8044000 comp rx=0 tx=0).stop 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39e55ef740 1 -- 172.21.15.39:0/3254147852 shutdown_connections 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39e55ef740 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d11b520 0x55985d4c00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39e55ef740 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d0fa840 0x55985d4bfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.145+0000 7f39e55ef740 1 --2- 172.21.15.39:0/3254147852 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0f2660 0x55985d4d9d10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.146+0000 7f39e55ef740 1 -- 172.21.15.39:0/3254147852 >> 172.21.15.39:0/3254147852 conn(0x55985d10f910 msgr2=0x55985d4cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.146+0000 7f39e55ef740 1 -- 172.21.15.39:0/3254147852 shutdown_connections 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.146+0000 7f39e55ef740 1 -- 172.21.15.39:0/3254147852 wait complete. 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.146+0000 7f39e55ef740 1 -- start start 2024-09-21T01:21:23.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.146+0000 7f39e55ef740 1 Processor -- start 2024-09-21T01:21:23.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 0x55985d4d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d11b520 0x55985d4d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81af3f0 con 0x55985d11b520 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81b0860 con 0x55985d0f2660 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39d81afac0 con 0x55985d0fa840 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39e55ef740 10 client.0 _ll_register_callbacks cb 0x7f39c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39d7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 0x55985d4d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39dca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d11b520 0x55985d4d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39d7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 0x55985d4d5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39572/0 (socket says 172.21.15.39:39572) 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39d7e00640 1 -- 172.21.15.39:0/713154069 learned_addr learned my addr 172.21.15.39:0/713154069 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.147+0000 7f39cf400640 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39d7e00640 1 -- 172.21.15.39:0/713154069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 msgr2=0x55985d4d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39d7e00640 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4d62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39d7e00640 1 -- 172.21.15.39:0/713154069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d11b520 msgr2=0x55985d4d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39d7e00640 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d11b520 0x55985d4d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39d7e00640 1 -- 172.21.15.39:0/713154069 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985d4e0060 con 0x55985d0f2660 2024-09-21T01:21:23.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39d7e00640 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 0x55985d4d5d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009600 tx=0x7f39c8031760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6efc84cb4d26829 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:23.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39c5600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.148+0000 7f39c5600640 1 -- 172.21.15.39:0/713154069 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c806b060 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39c8036050 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39c5600640 1 -- 172.21.15.39:0/713154069 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7f39c8004500 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39c5600640 1 -- 172.21.15.39:0/713154069 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f39c8004720 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39e55ef740 10 client.26261 fetch_fsmap learned FSMap version 55 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55985d54a6d0 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39c5600640 1 -- 172.21.15.39:0/713154069 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c8036050 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.149+0000 7f39c5600640 1 -- 172.21.15.39:0/713154069 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7f39c8003b70 con 0x55985d0f2660 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39e55ef740 10 client.26261 fetch_fsmap finished waiting for FSMap version 55 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39e55ef740 10 client.26261 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39e55ef740 20 client.26261 populate_metadata read hostname 'smithi039' 2024-09-21T01:21:23.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39e55ef740 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0af250 0x55985d54cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:23.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39e55ef740 4 client.26261 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c745c3b7", "release": true, "await": true}] 2024-09-21T01:21:23.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c745c3b7", "release": true, "await": true}) -- 0x55985d436c90 con 0x55985d0af250 2024-09-21T01:21:23.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.150+0000 7f39cf400640 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0af250 0x55985d54cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:23.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.151+0000 7f39cf400640 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0af250 0x55985d54cdc0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a45d8190dc571b0e server_cookie=935899dfee5c7d6f in_seq=0 out_seq=0 2024-09-21T01:21:23.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.151+0000 7f39c5600640 10 client.26261 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:23.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.181+0000 7f39c5600640 1 -- 172.21.15.39:0/713154069 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55985d436c90 con 0x55985d0af250 2024-09-21T01:21:23.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.181+0000 7f39c5600640 10 client.26261 handle_command_reply: tid=0 2024-09-21T01:21:23.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 client.26261 shutdown 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 msgr2=0x55985d4d5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 0x55985d4d5d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009600 tx=0x7f39c8031760 comp rx=0 tx=0).stop 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 shutdown_connections 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55985d0af250 0x55985d54cdc0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55985d11b520 0x55985d4d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55985d0fa840 0x55985d4d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 --2- 172.21.15.39:0/713154069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55985d0f2660 0x55985d4d5d80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.182+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 >> 172.21.15.39:0/713154069 conn(0x55985d10f910 msgr2=0x55985d4cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.183+0000 7f39c5600640 0 client.26261 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:23.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.183+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 shutdown_connections 2024-09-21T01:21:23.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.183+0000 7f39e55ef740 1 -- 172.21.15.39:0/713154069 wait complete. 2024-09-21T01:21:23.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.183+0000 7f39e55ef740 20 client.26261 trim_cache size 0 max 16384 2024-09-21T01:21:23.185 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:21:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 55, 2024-09-21T01:21:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:21:23.185 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 68, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "c745c3b7": { 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 68, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 130.0, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:21:23.186 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:21:23.187 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39bc0b6700 msgr2=0x7f39bc0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39bc0b6700 0x7f39bc0b8c10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55985d4c0fc0 tx=0x7f39c8009360 comp rx=0 tx=0).stop 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d8109c20 msgr2=0x7f39d819fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d8109c20 0x7f39d819fb10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f39d000e970 tx=0x7f39d000ee30 comp rx=0 tx=0).stop 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 shutdown_connections 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39d8109c20 0x7f39d819fb10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d81092d0 0x7f39d819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39bc0b6700 0x7f39bc0b8c10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 --2- 172.21.15.39:0/3916576305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39d8103990 0x7f39d819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.184+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 >> 172.21.15.39:0/3916576305 conn(0x7f39d80ff440 msgr2=0x7f39d8108980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.185+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 shutdown_connections 2024-09-21T01:21:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:23.185+0000 7f39dd400640 1 -- 172.21.15.39:0/3916576305 wait complete. 2024-09-21T01:21:23.772 INFO:tasks.workunit.client.0.smithi039.stdout: 10 62353 12.99 MB/sec execute 110 sec latency 8127.892 ms 2024-09-21T01:21:23.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:23 smithi160 ceph-mon[34696]: pgmap v949: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 295 op/s 2024-09-21T01:21:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:23 smithi039 ceph-mon[30415]: pgmap v949: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 295 op/s 2024-09-21T01:21:24.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:23 smithi150 ceph-mon[34945]: pgmap v949: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 295 op/s 2024-09-21T01:21:24.739 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:24 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:24.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 63036 13.08 MB/sec execute 111 sec latency 268.901 ms 2024-09-21T01:21:24.823 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c745c3b7', total seconds elapsed: 16.3 2024-09-21T01:21:24.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:24 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:25.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:24 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:25.616 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:25 smithi039 ceph-mon[30415]: pgmap v950: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 276 op/s 2024-09-21T01:21:25.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 63770 13.44 MB/sec execute 112 sec latency 74.375 ms 2024-09-21T01:21:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:25 smithi150 ceph-mon[34945]: pgmap v950: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 276 op/s 2024-09-21T01:21:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:25 smithi160 ceph-mon[34696]: pgmap v950: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 276 op/s 2024-09-21T01:21:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:21:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:21:26.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:21:26.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 64447 13.64 MB/sec execute 113 sec latency 67.805 ms 2024-09-21T01:21:27.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:21:27.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:21:27.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:21:27.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:21:27.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:21:27.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:21:27.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 65074 13.78 MB/sec execute 114 sec latency 76.114 ms 2024-09-21T01:21:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:27 smithi150 ceph-mon[34945]: pgmap v951: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-09-21T01:21:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:27 smithi160 ceph-mon[34696]: pgmap v951: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-09-21T01:21:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:27 smithi039 ceph-mon[30415]: pgmap v951: 129 pgs: 129 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-09-21T01:21:28.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 65948 14.06 MB/sec execute 115 sec latency 124.564 ms 2024-09-21T01:21:29.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 66462 14.22 MB/sec execute 116 sec latency 1124.617 ms 2024-09-21T01:21:29.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:29 smithi039 ceph-mon[30415]: pgmap v952: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.9 MiB/s wr, 347 op/s 2024-09-21T01:21:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:29 smithi150 ceph-mon[34945]: pgmap v952: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.9 MiB/s wr, 347 op/s 2024-09-21T01:21:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:29 smithi160 ceph-mon[34696]: pgmap v952: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.9 MiB/s wr, 347 op/s 2024-09-21T01:21:30.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 66977 14.34 MB/sec execute 117 sec latency 2124.666 ms 2024-09-21T01:21:31.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 67345 14.33 MB/sec execute 118 sec latency 3124.908 ms 2024-09-21T01:21:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:31 smithi150 ceph-mon[34945]: pgmap v953: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-09-21T01:21:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:31 smithi160 ceph-mon[34696]: pgmap v953: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-09-21T01:21:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:31 smithi039 ceph-mon[30415]: pgmap v953: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-09-21T01:21:32.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 67813 14.47 MB/sec execute 119 sec latency 4125.106 ms 2024-09-21T01:21:33.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 68224 14.54 MB/sec execute 120 sec latency 5125.164 ms 2024-09-21T01:21:34.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:33 smithi150 ceph-mon[34945]: pgmap v954: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.2 MiB/s wr, 294 op/s 2024-09-21T01:21:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:33 smithi160 ceph-mon[34696]: pgmap v954: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.2 MiB/s wr, 294 op/s 2024-09-21T01:21:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:33 smithi039 ceph-mon[30415]: pgmap v954: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.2 MiB/s wr, 294 op/s 2024-09-21T01:21:34.773 INFO:tasks.workunit.client.0.smithi039.stdout: 10 68610 14.54 MB/sec execute 121 sec latency 5994.544 ms 2024-09-21T01:21:35.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 69208 14.69 MB/sec execute 122 sec latency 6611.519 ms 2024-09-21T01:21:36.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:35 smithi150 ceph-mon[34945]: pgmap v955: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 377 op/s 2024-09-21T01:21:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:35 smithi160 ceph-mon[34696]: pgmap v955: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 377 op/s 2024-09-21T01:21:36.140 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:35 smithi039 ceph-mon[30415]: pgmap v955: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 377 op/s 2024-09-21T01:21:36.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 69840 14.90 MB/sec execute 123 sec latency 7611.631 ms 2024-09-21T01:21:37.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 70250 14.94 MB/sec execute 124 sec latency 8095.391 ms 2024-09-21T01:21:37.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:37 smithi039 ceph-mon[30415]: pgmap v956: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.3 MiB/s wr, 397 op/s 2024-09-21T01:21:38.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:37 smithi150 ceph-mon[34945]: pgmap v956: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.3 MiB/s wr, 397 op/s 2024-09-21T01:21:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:37 smithi160 ceph-mon[34696]: pgmap v956: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.3 MiB/s wr, 397 op/s 2024-09-21T01:21:38.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 70741 15.02 MB/sec execute 125 sec latency 8037.689 ms 2024-09-21T01:21:39.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 71214 15.08 MB/sec execute 126 sec latency 8768.777 ms 2024-09-21T01:21:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:39 smithi150 ceph-mon[34945]: pgmap v957: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.2 MiB/s wr, 471 op/s 2024-09-21T01:21:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:39 smithi160 ceph-mon[34696]: pgmap v957: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.2 MiB/s wr, 471 op/s 2024-09-21T01:21:40.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:39 smithi039 ceph-mon[30415]: pgmap v957: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.2 MiB/s wr, 471 op/s 2024-09-21T01:21:40.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 71861 15.23 MB/sec execute 127 sec latency 87.609 ms 2024-09-21T01:21:41.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 72598 15.44 MB/sec execute 128 sec latency 83.921 ms 2024-09-21T01:21:42.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:41 smithi150 ceph-mon[34945]: pgmap v958: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 6.4 MiB/s wr, 399 op/s 2024-09-21T01:21:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:41 smithi160 ceph-mon[34696]: pgmap v958: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 6.4 MiB/s wr, 399 op/s 2024-09-21T01:21:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:41 smithi039 ceph-mon[30415]: pgmap v958: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 6.4 MiB/s wr, 399 op/s 2024-09-21T01:21:42.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 73345 15.65 MB/sec execute 129 sec latency 106.171 ms 2024-09-21T01:21:43.395 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10.2 2024-09-21T01:21:43.396 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:21:43.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 73994 15.79 MB/sec execute 130 sec latency 109.616 ms 2024-09-21T01:21:43.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:43 smithi039 ceph-mon[30415]: pgmap v959: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.4 MiB/s wr, 341 op/s 2024-09-21T01:21:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:43 smithi150 ceph-mon[34945]: pgmap v959: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.4 MiB/s wr, 341 op/s 2024-09-21T01:21:44.086 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:21:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:43 smithi160 ceph-mon[34696]: pgmap v959: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.4 MiB/s wr, 341 op/s 2024-09-21T01:21:44.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 74525 15.87 MB/sec execute 131 sec latency 118.236 ms 2024-09-21T01:21:45.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 75101 15.98 MB/sec execute 132 sec latency 116.113 ms 2024-09-21T01:21:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:45 smithi039 ceph-mon[30415]: pgmap v960: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.6 MiB/s wr, 418 op/s 2024-09-21T01:21:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:45 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:21:45.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1963181502' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:46.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:45 smithi150 ceph-mon[34945]: pgmap v960: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.6 MiB/s wr, 418 op/s 2024-09-21T01:21:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:45 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:21:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1963181502' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:45 smithi160 ceph-mon[34696]: pgmap v960: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.6 MiB/s wr, 418 op/s 2024-09-21T01:21:46.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:45 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:21:46.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1963181502' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.091+0000 7facf3e00640 1 -- 172.21.15.39:0/182982464 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face80318a0 con 0x7facf4102440 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 -- 172.21.15.39:0/182982464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 msgr2=0x7facf4102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 --2- 172.21.15.39:0/182982464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 0x7facf4102850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7face8009900 tx=0x7face802a480 comp rx=0 tx=0).stop 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 -- 172.21.15.39:0/182982464 shutdown_connections 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 --2- 172.21.15.39:0/182982464 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4112410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 --2- 172.21.15.39:0/182982464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf4102d90 0x7facf410fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 --2- 172.21.15.39:0/182982464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 0x7facf4102850 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.092+0000 7facfb000640 1 -- 172.21.15.39:0/182982464 >> 172.21.15.39:0/182982464 conn(0x7facf40fe2c0 msgr2=0x7facf4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:46.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.093+0000 7facfb000640 1 -- 172.21.15.39:0/182982464 shutdown_connections 2024-09-21T01:21:46.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:21:46.302 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":56,"btime":"2024-09-21T01:21:44:738665+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:44.386909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:21:46.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.093+0000 7facfb000640 1 -- 172.21.15.39:0/182982464 wait complete. 2024-09-21T01:21:46.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.095+0000 7facfb000640 1 Processor -- start 2024-09-21T01:21:46.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.095+0000 7facfb000640 1 -- start start 2024-09-21T01:21:46.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.097+0000 7facfb000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 0x7facf410f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.097+0000 7facfb000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf4102d90 0x7facf410f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.097+0000 7facfb000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.097+0000 7facfb000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7facf4105e00 con 0x7facf4102d90 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.097+0000 7facfb000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7facf4105c80 con 0x7facf4102440 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.097+0000 7facfb000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7facf4105f80 con 0x7facf410fff0 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.098+0000 7facfa600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.098+0000 7facf9c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 0x7facf410f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.098+0000 7facf9200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf4102d90 0x7facf410f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.098+0000 7facfa600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:51070/0 (socket says 172.21.15.39:51070) 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.099+0000 7facfa600640 1 -- 172.21.15.39:0/1963181502 learned_addr learned my addr 172.21.15.39:0/1963181502 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.099+0000 7facfa600640 1 -- 172.21.15.39:0/1963181502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 msgr2=0x7facf410f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.099+0000 7facfa600640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 0x7facf410f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.100+0000 7facfa600640 1 -- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf4102d90 msgr2=0x7facf410f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.100+0000 7facfa600640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf4102d90 0x7facf410f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.100+0000 7facfa600640 1 -- 172.21.15.39:0/1963181502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf4106e90 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.100+0000 7facfa600640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4106820 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7facec00eaa0 tx=0x7facec00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.102+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec004d20 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.102+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facec005020 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.102+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec0116d0 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.102+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf4107180 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.102+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf410c050 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.106+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf406b500 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.108+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7facec011870 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.110+0000 7facf2000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7facdc0b6760 0x7facdc0b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.111+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7facec0e7c50 con 0x7facf410fff0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.111+0000 7facf9c00640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7facdc0b6760 0x7facdc0b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.112+0000 7facf9c00640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7facdc0b6760 0x7facdc0b8c70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7face8005ab0 tx=0x7face8000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:46.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.116+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7facec0ed020 con 0x7facf410fff0 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.527+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf410cb60 con 0x7facf410fff0 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.528+0000 7facf2000640 1 -- 172.21.15.39:0/1963181502 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11912 (secure 0 0 0) 0x7facec016030 con 0x7facf410fff0 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 56 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7facdc0b6760 msgr2=0x7facdc0b8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7facdc0b6760 0x7facdc0b8c70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7face8005ab0 tx=0x7face8000bc0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 msgr2=0x7facf4106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4106820 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7facec00eaa0 tx=0x7facec00eea0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 shutdown_connections 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7facf410fff0 0x7facf4106820 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf4102d90 0x7facf410f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7facdc0b6760 0x7facdc0b8c70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 --2- 172.21.15.39:0/1963181502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facf4102440 0x7facf410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.531+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 >> 172.21.15.39:0/1963181502 conn(0x7facf40fe2c0 msgr2=0x7facf4112130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.532+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 shutdown_connections 2024-09-21T01:21:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:45.532+0000 7facfb000640 1 -- 172.21.15.39:0/1963181502 wait complete. 2024-09-21T01:21:46.774 INFO:tasks.workunit.client.0.smithi039.stdout: 10 75731 16.13 MB/sec execute 133 sec latency 105.590 ms 2024-09-21T01:21:47.476 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 130.2 --await-for 45.0' 2024-09-21T01:21:47.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 130.2 --await-for 45.0 2024-09-21T01:21:47.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 75781 16.02 MB/sec execute 134 sec latency 810.927 ms 2024-09-21T01:21:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:47 smithi150 ceph-mon[34945]: pgmap v961: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 365 op/s 2024-09-21T01:21:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:47 smithi160 ceph-mon[34696]: pgmap v961: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 365 op/s 2024-09-21T01:21:48.123 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:21:48.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:47 smithi039 ceph-mon[30415]: pgmap v961: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 365 op/s 2024-09-21T01:21:48.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 75891 15.96 MB/sec execute 135 sec latency 1753.864 ms 2024-09-21T01:21:49.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 75892 15.84 MB/sec execute 136 sec latency 2753.908 ms 2024-09-21T01:21:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:49 smithi150 ceph-mon[34945]: pgmap v962: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.7 MiB/s wr, 467 op/s 2024-09-21T01:21:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:49 smithi160 ceph-mon[34696]: pgmap v962: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.7 MiB/s wr, 467 op/s 2024-09-21T01:21:50.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:49 smithi039 ceph-mon[30415]: pgmap v962: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.7 MiB/s wr, 467 op/s 2024-09-21T01:21:50.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 75892 15.73 MB/sec execute 137 sec latency 3753.950 ms 2024-09-21T01:21:51.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76003 15.67 MB/sec execute 138 sec latency 4753.996 ms 2024-09-21T01:21:51.978 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:51 smithi039 ceph-mon[30415]: pgmap v963: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 437 op/s 2024-09-21T01:21:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:51 smithi150 ceph-mon[34945]: pgmap v963: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 437 op/s 2024-09-21T01:21:52.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:51 smithi160 ceph-mon[34696]: pgmap v963: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 437 op/s 2024-09-21T01:21:52.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76012 15.56 MB/sec execute 139 sec latency 4902.637 ms 2024-09-21T01:21:53.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 15.46 MB/sec execute 140 sec latency 5836.063 ms 2024-09-21T01:21:53.977 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:53 smithi039 ceph-mon[30415]: pgmap v964: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 8.4 MiB/s wr, 407 op/s 2024-09-21T01:21:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:53 smithi150 ceph-mon[34945]: pgmap v964: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 8.4 MiB/s wr, 407 op/s 2024-09-21T01:21:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:53 smithi160 ceph-mon[34696]: pgmap v964: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 8.4 MiB/s wr, 407 op/s 2024-09-21T01:21:54.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 15.35 MB/sec execute 141 sec latency 6836.118 ms 2024-09-21T01:21:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 -- 172.21.15.39:0/2816745972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 msgr2=0x7fee800a28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 --2- 172.21.15.39:0/2816745972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee800a28c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fee880524b0 tx=0x7fee880981a0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 -- 172.21.15.39:0/2816745972 shutdown_connections 2024-09-21T01:21:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 --2- 172.21.15.39:0/2816745972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee800a28c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 --2- 172.21.15.39:0/2816745972 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee80097850 0x7fee8009fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 --2- 172.21.15.39:0/2816745972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee80096e20 0x7fee80097230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.803+0000 7fee8d200640 1 -- 172.21.15.39:0/2816745972 >> 172.21.15.39:0/2816745972 conn(0x7fee8000cd70 msgr2=0x7fee8000d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.804+0000 7fee8d200640 1 -- 172.21.15.39:0/2816745972 shutdown_connections 2024-09-21T01:21:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.804+0000 7fee8d200640 1 -- 172.21.15.39:0/2816745972 wait complete. 2024-09-21T01:21:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.805+0000 7fee8d200640 1 Processor -- start 2024-09-21T01:21:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.808+0000 7fee8d200640 1 -- start start 2024-09-21T01:21:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.809+0000 7fee8d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee80096e20 0x7fee80096740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.809+0000 7fee8d200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee80097850 0x7fee80094d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.809+0000 7fee8d200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee80095270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.809+0000 7fee8d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fee800a5ff0 con 0x7fee80096e20 2024-09-21T01:21:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.809+0000 7fee8d200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee800a5e70 con 0x7fee800a03d0 2024-09-21T01:21:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.809+0000 7fee8d200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fee800a6170 con 0x7fee80097850 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.810+0000 7fee86800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee80095270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.810+0000 7fee86800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee80095270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:49190/0 (socket says 172.21.15.39:49190) 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.810+0000 7fee86800640 1 -- 172.21.15.39:0/817208171 learned_addr learned my addr 172.21.15.39:0/817208171 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.811+0000 7fee86800640 1 -- 172.21.15.39:0/817208171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee80097850 msgr2=0x7fee80094d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.811+0000 7fee86800640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee80097850 0x7fee80094d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.811+0000 7fee86800640 1 -- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee80096e20 msgr2=0x7fee80096740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.811+0000 7fee86800640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee80096e20 0x7fee80096740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.811+0000 7fee86800640 1 -- 172.21.15.39:0/817208171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee80095a90 con 0x7fee800a03d0 2024-09-21T01:21:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.812+0000 7fee86800640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee80095270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fee88051d10 tx=0x7fee8806c290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.813+0000 7fee7ea00640 1 -- 172.21.15.39:0/817208171 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee880af070 con 0x7fee800a03d0 2024-09-21T01:21:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.813+0000 7fee7ea00640 1 -- 172.21.15.39:0/817208171 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8806a7e0 con 0x7fee800a03d0 2024-09-21T01:21:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.813+0000 7fee7ea00640 1 -- 172.21.15.39:0/817208171 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee880a16d0 con 0x7fee800a03d0 2024-09-21T01:21:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.813+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee80130ae0 con 0x7fee800a03d0 2024-09-21T01:21:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.815+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee80130fc0 con 0x7fee800a03d0 2024-09-21T01:21:54.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.822+0000 7fee7ea00640 1 -- 172.21.15.39:0/817208171 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fee880a1870 con 0x7fee800a03d0 2024-09-21T01:21:54.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.824+0000 7fee7ea00640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fee6c0b6700 0x7fee6c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.824+0000 7fee7ea00640 1 -- 172.21.15.39:0/817208171 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fee88174be0 con 0x7fee800a03d0 2024-09-21T01:21:54.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.824+0000 7fee93ed9740 1 Processor -- start 2024-09-21T01:21:54.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 -- start start 2024-09-21T01:21:54.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee85e00640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fee6c0b6700 0x7fee6c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 0x563ccdce1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd910980 0x563ccdce1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdce1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563ccdc56ca0 con 0x563ccd939840 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563ccdc460d0 con 0x563ccd910980 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee93ed9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563ccdc49ee0 con 0x563ccd918b60 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.825+0000 7fee85e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 0x563ccdce1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.826+0000 7fee85e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 0x563ccdce1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60222/0 (socket says 172.21.15.39:60222) 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.826+0000 7fee85e00640 1 -- 172.21.15.39:0/1121253185 learned_addr learned my addr 172.21.15.39:0/1121253185 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.826+0000 7fee85400640 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd910980 0x563ccdce1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.826+0000 7fee86800640 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdce1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.827+0000 7fee85e00640 1 -- 172.21.15.39:0/1121253185 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 msgr2=0x563ccdce1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.827+0000 7fee85e00640 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdce1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.827+0000 7fee85e00640 1 -- 172.21.15.39:0/1121253185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd910980 msgr2=0x563ccdce1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.827+0000 7fee85e00640 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd910980 0x563ccdce1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.827+0000 7fee85e00640 1 -- 172.21.15.39:0/1121253185 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ccdbf8400 con 0x563ccd939840 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.828+0000 7fee85e00640 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 0x563ccdce1510 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fee78011750 tx=0x7fee78019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad756f46bbe5fcfe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.828+0000 7fee85e00640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fee6c0b6700 0x7fee6c0b8c10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fee78014850 tx=0x7fee7800bf30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.829+0000 7fee73400640 1 -- 172.21.15.39:0/1121253185 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78011e50 con 0x563ccd939840 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.829+0000 7fee73400640 1 -- 172.21.15.39:0/1121253185 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee78024070 con 0x563ccd939840 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.829+0000 7fee73400640 1 -- 172.21.15.39:0/1121253185 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7801f6a0 con 0x563ccd939840 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 -- 172.21.15.39:0/1121253185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 msgr2=0x563ccdce1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 0x563ccdce1510 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fee78011750 tx=0x7fee78019000 comp rx=0 tx=0).stop 2024-09-21T01:21:54.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 -- 172.21.15.39:0/1121253185 shutdown_connections 2024-09-21T01:21:54.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdce1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd910980 0x563ccdce1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 --2- 172.21.15.39:0/1121253185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd939840 0x563ccdce1510 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.830+0000 7fee93ed9740 1 -- 172.21.15.39:0/1121253185 >> 172.21.15.39:0/1121253185 conn(0x563ccd92d990 msgr2=0x563ccdcdf400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:54.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.831+0000 7fee93ed9740 1 -- 172.21.15.39:0/1121253185 shutdown_connections 2024-09-21T01:21:54.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.831+0000 7fee93ed9740 1 -- 172.21.15.39:0/1121253185 wait complete. 2024-09-21T01:21:54.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.831+0000 7fee93ed9740 1 -- start start 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.832+0000 7fee93ed9740 1 Processor -- start 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.833+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcecbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.833+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdced110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdced650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee85e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcecbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee93ed9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563ccdaa77a0 con 0x563ccd910980 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee93ed9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563ccdbeba10 con 0x563ccd939840 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee93ed9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x563ccdcefc90 con 0x563ccd918b60 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee85400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdced110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee86800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdced650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee86800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdced650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:49212/0 (socket says 172.21.15.39:49212) 2024-09-21T01:21:54.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.834+0000 7fee86800640 1 -- 172.21.15.39:0/625482557 learned_addr learned my addr 172.21.15.39:0/625482557 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee86800640 1 -- 172.21.15.39:0/625482557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 msgr2=0x563ccdced110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee86800640 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdced110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee86800640 1 -- 172.21.15.39:0/625482557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 msgr2=0x563ccdcecbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee86800640 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcecbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee86800640 1 -- 172.21.15.39:0/625482557 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ccd9b3b90 con 0x563ccd939840 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee86800640 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdced650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee88069880 tx=0x7fee880a1f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de147a9dacadbff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee67e00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.835+0000 7fee85400640 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdced110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee93ed9740 10 client.0 _ll_register_callbacks cb 0x563ccdbfbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:21:54.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee78024070 con 0x563ccd939840 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee67e00640 1 -- 172.21.15.39:0/625482557 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88139dc0 con 0x563ccd939840 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee67e00640 1 -- 172.21.15.39:0/625482557 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fee881361b0 con 0x563ccd939840 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee67e00640 1 -- 172.21.15.39:0/625482557 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fee88139dc0 con 0x563ccd939840 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee93ed9740 10 client.26279 fetch_fsmap learned FSMap version 56 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.836+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({fsmap=56}) -- 0x563ccdcfb570 con 0x563ccd939840 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee67e00640 1 -- 172.21.15.39:0/625482557 <== mon.2 v2:172.21.15.150:3300/0 4 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fee88139320 con 0x563ccd939840 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee93ed9740 10 client.26279 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee93ed9740 10 client.26279 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee93ed9740 20 client.26279 populate_metadata read hostname 'smithi039' 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee93ed9740 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccd8f81c0 0x563ccdcfed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee93ed9740 4 client.26279 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:21:54.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.837+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ccdc58bf0 con 0x563ccd8f81c0 2024-09-21T01:21:54.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.854+0000 7fee85e00640 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccd8f81c0 0x563ccdcfed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.890+0000 7fee85e00640 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccd8f81c0 0x563ccdcfed10 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a3dfb8eade1f73 server_cookie=da0b92eaf38a2914 in_seq=0 out_seq=0 2024-09-21T01:21:54.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.890+0000 7fee67e00640 10 client.26279 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:54.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.901+0000 7fee67e00640 1 -- 172.21.15.39:0/625482557 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563ccdc58bf0 con 0x563ccd8f81c0 2024-09-21T01:21:54.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.901+0000 7fee67e00640 10 client.26279 handle_command_reply: tid=0 2024-09-21T01:21:54.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.902+0000 7fee93ed9740 1 client.26279 shutdown 2024-09-21T01:21:54.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.902+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 msgr2=0x563ccdced650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:54.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.902+0000 7fee93ed9740 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdced650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee88069880 tx=0x7fee880a1f30 comp rx=0 tx=0).stop 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 shutdown_connections 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccd8f81c0 0x563ccdcfed10 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdced650 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd918b60 0x563ccdced110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 --2- 172.21.15.39:0/625482557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcecbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 >> 172.21.15.39:0/625482557 conn(0x563ccd92d990 msgr2=0x563ccdce46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:54.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee67e00640 0 client.26279 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:54.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 shutdown_connections 2024-09-21T01:21:54.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.906+0000 7fee93ed9740 1 -- 172.21.15.39:0/625482557 wait complete. 2024-09-21T01:21:54.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.907+0000 7fee93ed9740 20 client.26279 trim_cache size 0 max 16384 2024-09-21T01:21:54.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.945+0000 7fee93ed9740 1 Processor -- start 2024-09-21T01:21:54.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.945+0000 7fee93ed9740 1 -- start start 2024-09-21T01:21:54.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.946+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd910980 0x563ccdce2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.946+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 0x563ccdcf8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.946+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.946+0000 7fee93ed9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fee80154af0 con 0x563ccd918b60 2024-09-21T01:21:54.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.946+0000 7fee93ed9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee80152390 con 0x563ccd939840 2024-09-21T01:21:54.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.946+0000 7fee93ed9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fee80096030 con 0x563ccd910980 2024-09-21T01:21:54.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.947+0000 7fee85400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 0x563ccdcf8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.947+0000 7fee85400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 0x563ccdcf8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60232/0 (socket says 172.21.15.39:60232) 2024-09-21T01:21:54.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.947+0000 7fee85400640 1 -- 172.21.15.39:0/448631531 learned_addr learned my addr 172.21.15.39:0/448631531 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:54.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee86800640 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee85400640 1 -- 172.21.15.39:0/448631531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd910980 msgr2=0x563ccdce2f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:21:54.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee85400640 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd910980 0x563ccdce2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee85400640 1 -- 172.21.15.39:0/448631531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 msgr2=0x563ccdcf8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:54.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee85400640 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee85400640 1 -- 172.21.15.39:0/448631531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ccdceefa0 con 0x563ccd918b60 2024-09-21T01:21:54.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.948+0000 7fee86800640 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:21:54.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.949+0000 7fee85400640 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 0x563ccdcf8660 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fee7401a9e0 tx=0x7fee7401aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54f9ef3e75e39d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:54.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.951+0000 7fee72a00640 1 -- 172.21.15.39:0/448631531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7402a020 con 0x563ccd918b60 2024-09-21T01:21:54.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.951+0000 7fee72a00640 1 -- 172.21.15.39:0/448631531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee74020bf0 con 0x563ccd918b60 2024-09-21T01:21:54.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee72a00640 1 -- 172.21.15.39:0/448631531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7401f040 con 0x563ccd918b60 2024-09-21T01:21:54.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 -- 172.21.15.39:0/448631531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 msgr2=0x563ccdcf8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 0x563ccdcf8660 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fee7401a9e0 tx=0x7fee7401aea0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 -- 172.21.15.39:0/448631531 shutdown_connections 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd918b60 0x563ccdcf8660 secure :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fee7401a9e0 tx=0x7fee7401aea0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 --2- 172.21.15.39:0/448631531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd910980 0x563ccdce2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.952+0000 7fee93ed9740 1 -- 172.21.15.39:0/448631531 >> 172.21.15.39:0/448631531 conn(0x563ccd92d990 msgr2=0x563ccdcf7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:21:54.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.955+0000 7fee93ed9740 1 -- 172.21.15.39:0/448631531 shutdown_connections 2024-09-21T01:21:54.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.955+0000 7fee93ed9740 1 -- 172.21.15.39:0/448631531 wait complete. 2024-09-21T01:21:54.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.956+0000 7fee93ed9740 1 -- start start 2024-09-21T01:21:54.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.956+0000 7fee93ed9740 1 Processor -- start 2024-09-21T01:21:54.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.957+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcf1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.957+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee85e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcf1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee93ed9740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd8cd570 0x563ccdd62750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee93ed9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fee80138f80 con 0x563ccd910980 2024-09-21T01:21:54.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee93ed9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee80151cc0 con 0x563ccd939840 2024-09-21T01:21:54.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee93ed9740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fee80150f20 con 0x563ccd8cd570 2024-09-21T01:21:54.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee93ed9740 10 client.0 _ll_register_callbacks cb 0x7fee88136af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:21:54.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee85e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcf1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60234/0 (socket says 172.21.15.39:60234) 2024-09-21T01:21:54.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.958+0000 7fee85e00640 1 -- 172.21.15.39:0/3981638094 learned_addr learned my addr 172.21.15.39:0/3981638094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:21:54.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.959+0000 7fee85400640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:54.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.959+0000 7fee85e00640 1 -- 172.21.15.39:0/3981638094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd8cd570 msgr2=0x563ccdd62750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:21:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.959+0000 7fee85e00640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd8cd570 0x563ccdd62750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.959+0000 7fee85e00640 1 -- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 msgr2=0x563ccdcf1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:21:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.959+0000 7fee85e00640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:21:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.959+0000 7fee85e00640 1 -- 172.21.15.39:0/3981638094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ccdd62dc0 con 0x563ccd910980 2024-09-21T01:21:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.960+0000 7fee85400640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:21:54.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.960+0000 7fee85e00640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcf1900 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee7800fe60 tx=0x7fee7800bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3ae3ccf5991613 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:21:54.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.961+0000 7fee67e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:21:54.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.961+0000 7fee67e00640 1 -- 172.21.15.39:0/3981638094 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78011e50 con 0x563ccd910980 2024-09-21T01:21:54.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.962+0000 7fee67e00640 1 -- 172.21.15.39:0/3981638094 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fee780140b0 con 0x563ccd910980 2024-09-21T01:21:54.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.962+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee74020bf0 con 0x563ccd910980 2024-09-21T01:21:54.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.963+0000 7fee67e00640 1 -- 172.21.15.39:0/3981638094 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fee7803a050 con 0x563ccd910980 2024-09-21T01:21:54.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.964+0000 7fee93ed9740 10 client.16257 fetch_fsmap learned FSMap version 56 2024-09-21T01:21:54.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.964+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x563ccdd6a030 con 0x563ccd910980 2024-09-21T01:21:54.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.965+0000 7fee67e00640 1 -- 172.21.15.39:0/3981638094 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee74020bf0 con 0x563ccd910980 2024-09-21T01:21:54.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.965+0000 7fee67e00640 1 -- 172.21.15.39:0/3981638094 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fee7801f780 con 0x563ccd910980 2024-09-21T01:21:54.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.966+0000 7fee93ed9740 10 client.16257 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:21:54.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.966+0000 7fee93ed9740 10 client.16257 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:21:54.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.966+0000 7fee93ed9740 20 client.16257 populate_metadata read hostname 'smithi039' 2024-09-21T01:21:54.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.966+0000 7fee93ed9740 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccdd6a4f0 0x563ccdd6c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:21:54.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.966+0000 7fee93ed9740 4 client.16257 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.2, "await_for": 45.0}] 2024-09-21T01:21:54.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.966+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.2, "await_for": 45.0}) -- 0x563ccdc546a0 con 0x563ccdd6a4f0 2024-09-21T01:21:54.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:54.985+0000 7fee85400640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccdd6a4f0 0x563ccdd6c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:21:55.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:55.044+0000 7fee67e00640 10 client.16257 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:21:55.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:21:55.044+0000 7fee85400640 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccdd6a4f0 0x563ccdd6c910 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f479dc2786e5a6d3 server_cookie=48da5ea04d8c8b93 in_seq=0 out_seq=0 2024-09-21T01:21:55.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 15.24 MB/sec execute 142 sec latency 7836.171 ms 2024-09-21T01:21:55.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:55 smithi039 ceph-mon[30415]: pgmap v965: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 11 MiB/s wr, 458 op/s 2024-09-21T01:21:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:55 smithi150 ceph-mon[34945]: pgmap v965: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 11 MiB/s wr, 458 op/s 2024-09-21T01:21:56.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:55 smithi160 ceph-mon[34696]: pgmap v965: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 11 MiB/s wr, 458 op/s 2024-09-21T01:21:56.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 15.13 MB/sec execute 143 sec latency 8836.222 ms 2024-09-21T01:21:57.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 15.03 MB/sec execute 144 sec latency 9836.274 ms 2024-09-21T01:21:57.900 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:57 smithi039 ceph-mon[30415]: pgmap v966: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 381 op/s 2024-09-21T01:21:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:57 smithi150 ceph-mon[34945]: pgmap v966: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 381 op/s 2024-09-21T01:21:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:57 smithi160 ceph-mon[34696]: pgmap v966: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 381 op/s 2024-09-21T01:21:58.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.92 MB/sec execute 145 sec latency 10836.329 ms 2024-09-21T01:21:59.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.82 MB/sec execute 146 sec latency 11836.387 ms 2024-09-21T01:21:59.881 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:21:59 smithi039 ceph-mon[30415]: pgmap v967: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 14 MiB/s wr, 487 op/s 2024-09-21T01:22:00.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:21:59 smithi160 ceph-mon[34696]: pgmap v967: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 14 MiB/s wr, 487 op/s 2024-09-21T01:22:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:21:59 smithi150 ceph-mon[34945]: pgmap v967: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 14 MiB/s wr, 487 op/s 2024-09-21T01:22:00.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.72 MB/sec execute 147 sec latency 12836.438 ms 2024-09-21T01:22:01.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.62 MB/sec execute 148 sec latency 13836.495 ms 2024-09-21T01:22:02.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:01 smithi160 ceph-mon[34696]: pgmap v968: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 10 MiB/s wr, 367 op/s 2024-09-21T01:22:02.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:01 smithi039 ceph-mon[30415]: pgmap v968: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 10 MiB/s wr, 367 op/s 2024-09-21T01:22:02.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:01 smithi150 ceph-mon[34945]: pgmap v968: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 10 MiB/s wr, 367 op/s 2024-09-21T01:22:02.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.52 MB/sec execute 149 sec latency 14836.545 ms 2024-09-21T01:22:03.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.43 MB/sec execute 150 sec latency 15836.601 ms 2024-09-21T01:22:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:03 smithi160 ceph-mon[34696]: pgmap v969: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.0 MiB/s wr, 304 op/s 2024-09-21T01:22:04.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:03 smithi039 ceph-mon[30415]: pgmap v969: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.0 MiB/s wr, 304 op/s 2024-09-21T01:22:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:03 smithi150 ceph-mon[34945]: pgmap v969: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.0 MiB/s wr, 304 op/s 2024-09-21T01:22:04.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.33 MB/sec execute 151 sec latency 16836.655 ms 2024-09-21T01:22:05.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.24 MB/sec execute 152 sec latency 17836.710 ms 2024-09-21T01:22:06.025 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:05 smithi039 ceph-mon[30415]: pgmap v970: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.6 MiB/s wr, 291 op/s 2024-09-21T01:22:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:05 smithi160 ceph-mon[34696]: pgmap v970: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.6 MiB/s wr, 291 op/s 2024-09-21T01:22:06.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:05 smithi150 ceph-mon[34945]: pgmap v970: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.6 MiB/s wr, 291 op/s 2024-09-21T01:22:06.775 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76043 14.14 MB/sec execute 153 sec latency 18513.343 ms 2024-09-21T01:22:07.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76044 14.05 MB/sec execute 154 sec latency 16031.619 ms 2024-09-21T01:22:07.974 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:07 smithi039 ceph-mon[30415]: pgmap v971: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.6 MiB/s wr, 209 op/s 2024-09-21T01:22:08.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:07 smithi150 ceph-mon[34945]: pgmap v971: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.6 MiB/s wr, 209 op/s 2024-09-21T01:22:08.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:07 smithi160 ceph-mon[34696]: pgmap v971: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.6 MiB/s wr, 209 op/s 2024-09-21T01:22:08.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76044 13.96 MB/sec execute 155 sec latency 17031.665 ms 2024-09-21T01:22:09.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76044 13.87 MB/sec execute 156 sec latency 18031.712 ms 2024-09-21T01:22:10.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:09 smithi039 ceph-mon[30415]: pgmap v972: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 225 op/s 2024-09-21T01:22:10.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.291+0000 7fee67e00640 1 -- 172.21.15.39:0/3981638094 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563ccdc546a0 con 0x563ccdd6a4f0 2024-09-21T01:22:10.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.291+0000 7fee67e00640 10 client.16257 handle_command_reply: tid=0 2024-09-21T01:22:10.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.291+0000 7fee93ed9740 1 client.16257 shutdown 2024-09-21T01:22:10.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 msgr2=0x563ccdcf1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:10.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcf1900 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee7800fe60 tx=0x7fee7800bfa0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 shutdown_connections 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x563ccdd6a4f0 0x563ccdd6c910 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x563ccd8cd570 0x563ccdd62750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ccd939840 0x563ccdcf1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 --2- 172.21.15.39:0/3981638094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ccd910980 0x563ccdcf1900 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 >> 172.21.15.39:0/3981638094 conn(0x563ccd92d990 msgr2=0x563ccdce2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee67e00640 0 client.16257 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 shutdown_connections 2024-09-21T01:22:10.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.292+0000 7fee93ed9740 1 -- 172.21.15.39:0/3981638094 wait complete. 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.293+0000 7fee93ed9740 20 client.16257 trim_cache size 0 max 16384 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 56, 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 70, 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout: "eef6e4dc": { 2024-09-21T01:22:10.295 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 70, 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 130.2, 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:10.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.294+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fee6c0b6700 msgr2=0x7fee6c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.294+0000 7fee8d200640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fee6c0b6700 0x7fee6c0b8c10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fee78014850 tx=0x7fee7800bf30 comp rx=0 tx=0).stop 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.294+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 msgr2=0x7fee80095270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:10.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.294+0000 7fee8d200640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee80095270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fee88051d10 tx=0x7fee8806c290 comp rx=0 tx=0).stop 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 shutdown_connections 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fee6c0b6700 0x7fee6c0b8c10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee800a03d0 0x7fee80095270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee80097850 0x7fee80094d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 --2- 172.21.15.39:0/817208171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee80096e20 0x7fee80096740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 >> 172.21.15.39:0/817208171 conn(0x7fee8000cd70 msgr2=0x7fee80099ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.296+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 shutdown_connections 2024-09-21T01:22:10.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:10.297+0000 7fee8d200640 1 -- 172.21.15.39:0/817208171 wait complete. 2024-09-21T01:22:10.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:09 smithi150 ceph-mon[34945]: pgmap v972: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 225 op/s 2024-09-21T01:22:10.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:09 smithi160 ceph-mon[34696]: pgmap v972: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 225 op/s 2024-09-21T01:22:10.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76061 13.78 MB/sec execute 157 sec latency 18140.674 ms 2024-09-21T01:22:11.237 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'eef6e4dc', quiesce took 27.8 seconds. Will release after: -17.6 2024-09-21T01:22:11.238 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:22:11.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76061 13.70 MB/sec execute 158 sec latency 5316.185 ms 2024-09-21T01:22:12.079 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:11 smithi039 ceph-mon[30415]: pgmap v973: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 700 KiB/s wr, 93 op/s 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.263+0000 7f01c9a00640 1 -- 172.21.15.39:0/409165624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410b510 msgr2=0x7f01c410b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.263+0000 7f01c9a00640 1 --2- 172.21.15.39:0/409165624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410b510 0x7f01c410b920 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009900 tx=0x7f01b002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.263+0000 7f01c9a00640 1 -- 172.21.15.39:0/409165624 shutdown_connections 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.263+0000 7f01c9a00640 1 --2- 172.21.15.39:0/409165624 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01c410be60 0x7f01c4112690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.263+0000 7f01c9a00640 1 --2- 172.21.15.39:0/409165624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410b510 0x7f01c410b920 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.263+0000 7f01c9a00640 1 --2- 172.21.15.39:0/409165624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c4105bd0 0x7f01c4105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.264+0000 7f01c9a00640 1 -- 172.21.15.39:0/409165624 >> 172.21.15.39:0/409165624 conn(0x7f01c4101620 msgr2=0x7f01c4103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.264+0000 7f01c9a00640 1 -- 172.21.15.39:0/409165624 shutdown_connections 2024-09-21T01:22:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.264+0000 7f01c9a00640 1 -- 172.21.15.39:0/409165624 wait complete. 2024-09-21T01:22:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 Processor -- start 2024-09-21T01:22:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 -- start start 2024-09-21T01:22:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01c4105bd0 0x7f01c419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c410b510 0x7f01c419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 0x7f01c41a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f01c4115b20 con 0x7f01c410be60 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01c41159a0 con 0x7f01c410b510 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.265+0000 7f01c9a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f01c4115ca0 con 0x7f01c4105bd0 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 0x7f01c41a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 0x7f01c41a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52972/0 (socket says 172.21.15.39:52972) 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 -- 172.21.15.39:0/3382667529 learned_addr learned my addr 172.21.15.39:0/3382667529 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c3e00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01c4105bd0 0x7f01c419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c3400640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c410b510 0x7f01c419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 -- 172.21.15.39:0/3382667529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01c4105bd0 msgr2=0x7f01c419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01c4105bd0 0x7f01c419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 -- 172.21.15.39:0/3382667529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c410b510 msgr2=0x7f01c419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c410b510 0x7f01c419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.266+0000 7f01c9000640 1 -- 172.21.15.39:0/3382667529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c41a2460 con 0x7f01c410be60 2024-09-21T01:22:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.267+0000 7f01c9000640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 0x7f01c41a1d30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00eaa0 tx=0x7f01bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.267+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc00cdb0 con 0x7f01c410be60 2024-09-21T01:22:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.267+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c41a2750 con 0x7f01c410be60 2024-09-21T01:22:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.267+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01bc005020 con 0x7f01c410be60 2024-09-21T01:22:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.267+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0116d0 con 0x7f01c410be60 2024-09-21T01:22:12.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.267+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c41a8eb0 con 0x7f01c410be60 2024-09-21T01:22:12.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.268+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c4105fe0 con 0x7f01c410be60 2024-09-21T01:22:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.271+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f01bc0054a0 con 0x7f01c410be60 2024-09-21T01:22:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.273+0000 7f01c0c00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01a80b6680 0x7f01a80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.274+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7f01bc0e8060 con 0x7f01c410be60 2024-09-21T01:22:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.274+0000 7f01c3e00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01a80b6680 0x7f01a80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.275+0000 7f01c3e00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01a80b6680 0x7f01a80b8b90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f01b400bbe0 tx=0x7f01b400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.276+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f01bc016030 con 0x7f01c410be60 2024-09-21T01:22:12.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:11 smithi150 ceph-mon[34945]: pgmap v973: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 700 KiB/s wr, 93 op/s 2024-09-21T01:22:12.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:11 smithi160 ceph-mon[34696]: pgmap v973: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 700 KiB/s wr, 93 op/s 2024-09-21T01:22:12.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.514+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01c419d620 con 0x7f01c410be60 2024-09-21T01:22:12.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.514+0000 7f01c0c00640 1 -- 172.21.15.39:0/3382667529 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11912 (secure 0 0 0) 0x7f01c419d620 con 0x7f01c410be60 2024-09-21T01:22:12.516 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:22:12.516 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":56,"btime":"2024-09-21T01:21:44:738665+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:44.386909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:22:12.516 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 56 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01a80b6680 msgr2=0x7f01a80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01a80b6680 0x7f01a80b8b90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f01b400bbe0 tx=0x7f01b400b390 comp rx=0 tx=0).stop 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 msgr2=0x7f01c41a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 0x7f01c41a1d30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00eaa0 tx=0x7f01bc00ef60 comp rx=0 tx=0).stop 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 shutdown_connections 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c410be60 0x7f01c41a1d30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f01a80b6680 0x7f01a80b8b90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c410b510 0x7f01c419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 --2- 172.21.15.39:0/3382667529 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01c4105bd0 0x7f01c419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.517+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 >> 172.21.15.39:0/3382667529 conn(0x7f01c4101620 msgr2=0x7f01c4103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.518+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 shutdown_connections 2024-09-21T01:22:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:12.518+0000 7f01c9a00640 1 -- 172.21.15.39:0/3382667529 wait complete. 2024-09-21T01:22:12.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76061 13.61 MB/sec execute 159 sec latency 6316.235 ms 2024-09-21T01:22:13.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:12 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3382667529' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:22:13.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:12 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3382667529' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:22:13.339 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id eef6e4dc --release --await' 2024-09-21T01:22:13.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id eef6e4dc --release --await 2024-09-21T01:22:13.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:12 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3382667529' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:22:13.731 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:22:13.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76061 13.53 MB/sec execute 160 sec latency 7316.279 ms 2024-09-21T01:22:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:13 smithi039 ceph-mon[30415]: pgmap v974: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 760 KiB/s wr, 92 op/s 2024-09-21T01:22:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:13 smithi150 ceph-mon[34945]: pgmap v974: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 760 KiB/s wr, 92 op/s 2024-09-21T01:22:14.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:13 smithi160 ceph-mon[34696]: pgmap v974: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 760 KiB/s wr, 92 op/s 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.412+0000 7fe3a3200640 1 -- 172.21.15.39:0/3021098029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a41092f0 msgr2=0x7fe3a4109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.412+0000 7fe3a3200640 1 --2- 172.21.15.39:0/3021098029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a41092f0 0x7fe3a4109700 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe390009900 tx=0x7fe39002a410 comp rx=0 tx=0).stop 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.412+0000 7fe3a3200640 1 -- 172.21.15.39:0/3021098029 shutdown_connections 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.412+0000 7fe3a3200640 1 --2- 172.21.15.39:0/3021098029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a4109c40 0x7fe3a4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.413+0000 7fe3a3200640 1 --2- 172.21.15.39:0/3021098029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a41092f0 0x7fe3a4109700 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.413+0000 7fe3a3200640 1 --2- 172.21.15.39:0/3021098029 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3a41039b0 0x7fe3a4103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.413+0000 7fe3a3200640 1 -- 172.21.15.39:0/3021098029 >> 172.21.15.39:0/3021098029 conn(0x7fe3a40ff400 msgr2=0x7fe3a4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:14.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.413+0000 7fe3a3200640 1 -- 172.21.15.39:0/3021098029 shutdown_connections 2024-09-21T01:22:14.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.413+0000 7fe3a3200640 1 -- 172.21.15.39:0/3021098029 wait complete. 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 Processor -- start 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 -- start start 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a41039b0 0x7fe3a419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3a41092f0 0x7fe3a419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 0x7fe3a41a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4113950 con 0x7fe3a4109c40 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41137d0 con 0x7fe3a41039b0 2024-09-21T01:22:14.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.414+0000 7fe3a3200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4113ad0 con 0x7fe3a41092f0 2024-09-21T01:22:14.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 0x7fe3a41a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 0x7fe3a41a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53876/0 (socket says 172.21.15.39:53876) 2024-09-21T01:22:14.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 -- 172.21.15.39:0/2306642665 learned_addr learned my addr 172.21.15.39:0/2306642665 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:14.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a1400640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3a41092f0 0x7fe3a419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 -- 172.21.15.39:0/2306642665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3a41092f0 msgr2=0x7fe3a419cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3a41092f0 0x7fe3a419cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 -- 172.21.15.39:0/2306642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a41039b0 msgr2=0x7fe3a419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a41039b0 0x7fe3a419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.415+0000 7fe3a2800640 1 -- 172.21.15.39:0/2306642665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a41a2510 con 0x7fe3a4109c40 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.416+0000 7fe3a2800640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 0x7fe3a41a1de0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00eb10 tx=0x7fe39c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.416+0000 7fe39aa00640 1 -- 172.21.15.39:0/2306642665 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c004b20 con 0x7fe3a4109c40 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.416+0000 7fe39aa00640 1 -- 172.21.15.39:0/2306642665 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe39c004cc0 con 0x7fe3a4109c40 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.416+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a41a2800 con 0x7fe3a4109c40 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.416+0000 7fe39aa00640 1 -- 172.21.15.39:0/2306642665 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c010670 con 0x7fe3a4109c40 2024-09-21T01:22:14.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.416+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a41a8f60 con 0x7fe3a4109c40 2024-09-21T01:22:14.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.419+0000 7fe3afc75740 1 Processor -- start 2024-09-21T01:22:14.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.420+0000 7fe39aa00640 1 -- 172.21.15.39:0/2306642665 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fe39c005020 con 0x7fe3a4109c40 2024-09-21T01:22:14.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.420+0000 7fe3afc75740 1 -- start start 2024-09-21T01:22:14.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.420+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x5600990fe340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.420+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x5600990fe880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.420+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x5600990fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3afc75740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560099075290 con 0x560098d2e660 2024-09-21T01:22:14.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3afc75740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5600990646c0 con 0x560098d57520 2024-09-21T01:22:14.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3afc75740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x5600990684d0 con 0x560098d36840 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x5600990fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x5600990fe880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53886/0 (socket says 172.21.15.39:53886) 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 -- 172.21.15.39:0/2990497104 learned_addr learned my addr 172.21.15.39:0/2990497104 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1400640 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x5600990fe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1400640 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x5600990fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 -- 172.21.15.39:0/2990497104 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 msgr2=0x5600990fedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x5600990fedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 -- 172.21.15.39:0/2990497104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 msgr2=0x5600990fe340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x5600990fe340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.421+0000 7fe3a1e00640 1 -- 172.21.15.39:0/2990497104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560098ff1460 con 0x560098d2e660 2024-09-21T01:22:14.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x5600990fe880 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe394009c00 tx=0x7fe39400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90aa3706eb9a19c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:14.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe38f400640 1 -- 172.21.15.39:0/2990497104 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394004030 con 0x560098d2e660 2024-09-21T01:22:14.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe38f400640 1 -- 172.21.15.39:0/2990497104 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3940035f0 con 0x560098d2e660 2024-09-21T01:22:14.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe39aa00640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3880b6680 0x7fe3880b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe38f400640 1 -- 172.21.15.39:0/2990497104 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394010670 con 0x560098d2e660 2024-09-21T01:22:14.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe39aa00640 1 -- 172.21.15.39:0/2306642665 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fe39c0e8190 con 0x7fe3a4109c40 2024-09-21T01:22:14.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.422+0000 7fe3a1400640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3880b6680 0x7fe3880b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 -- 172.21.15.39:0/2990497104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 msgr2=0x5600990fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x5600990fe880 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe394009c00 tx=0x7fe39400e720 comp rx=0 tx=0).stop 2024-09-21T01:22:14.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3a1400640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3880b6680 0x7fe3880b8b90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe39002aa00 tx=0x7fe390041040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:14.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 -- 172.21.15.39:0/2990497104 shutdown_connections 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x5600990fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x5600990fe880 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2990497104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x5600990fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 -- 172.21.15.39:0/2990497104 >> 172.21.15.39:0/2990497104 conn(0x560098d4b910 msgr2=0x5600990fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 -- 172.21.15.39:0/2990497104 shutdown_connections 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.423+0000 7fe3afc75740 1 -- 172.21.15.39:0/2990497104 wait complete. 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.424+0000 7fe3afc75740 1 -- start start 2024-09-21T01:22:14.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.424+0000 7fe3afc75740 1 Processor -- start 2024-09-21T01:22:14.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x560099109f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x56009910a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x560099115e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560098ec5bb0 con 0x560098d2e660 2024-09-21T01:22:14.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560098fff6c0 con 0x560098d57520 2024-09-21T01:22:14.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x56009910cb20 con 0x560098d36840 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3afc75740 10 client.0 _ll_register_callbacks cb 0x56009901a490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x560099109f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x560099109f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53892/0 (socket says 172.21.15.39:53892) 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3a1e00640 1 -- 172.21.15.39:0/3177363651 learned_addr learned my addr 172.21.15.39:0/3177363651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3a2800640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x560099115e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.425+0000 7fe3a1400640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x56009910a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a2800640 1 -- 172.21.15.39:0/3177363651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 msgr2=0x56009910a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a2800640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x56009910a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a2800640 1 -- 172.21.15.39:0/3177363651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 msgr2=0x560099109f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a2800640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x560099109f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a2800640 1 -- 172.21.15.39:0/3177363651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560098dc2b00 con 0x560098d57520 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x560099109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3a2800640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x560099115e40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0e54a0 tx=0x7fe39c0a9b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dff6485fc445337 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:14.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe383e00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe383e00640 1 -- 172.21.15.39:0/3177363651 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0ad310 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe383e00640 1 -- 172.21.15.39:0/3177363651 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fe39c0cfbb0 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3940035f0 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.426+0000 7fe383e00640 1 -- 172.21.15.39:0/3177363651 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fe39c0ad6b0 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 10 client.26294 fetch_fsmap learned FSMap version 56 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x560098e39960 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe383e00640 1 -- 172.21.15.39:0/3177363651 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3940035f0 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe383e00640 1 -- 172.21.15.39:0/3177363651 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fe39c0b21c0 con 0x560098d57520 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 10 client.26294 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 10 client.26294 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 20 client.26294 populate_metadata read hostname 'smithi039' 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098d15ea0 0x56009911bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 4 client.26294 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:22:14.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600990771e0 con 0x560098d15ea0 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.427+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098d15ea0 0x56009911bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.428+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098d15ea0 0x56009911bf10 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dec1b6038430e36 server_cookie=558960dd8648b6dd in_seq=0 out_seq=0 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.428+0000 7fe383e00640 10 client.26294 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe383e00640 1 -- 172.21.15.39:0/3177363651 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5600990771e0 con 0x560098d15ea0 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe383e00640 10 client.26294 handle_command_reply: tid=0 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 client.26294 shutdown 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 msgr2=0x560099115e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x560099115e40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0e54a0 tx=0x7fe39c0a9b70 comp rx=0 tx=0).stop 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 shutdown_connections 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098d15ea0 0x56009911bf10 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d57520 0x560099115e40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x56009910a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 --2- 172.21.15.39:0/3177363651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d2e660 0x560099109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe383e00640 0 client.26294 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:14.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 >> 172.21.15.39:0/3177363651 conn(0x560098d4b910 msgr2=0x5600991014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:14.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.429+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 shutdown_connections 2024-09-21T01:22:14.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.430+0000 7fe3afc75740 1 -- 172.21.15.39:0/3177363651 wait complete. 2024-09-21T01:22:14.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.430+0000 7fe3afc75740 20 client.26294 trim_cache size 0 max 16384 2024-09-21T01:22:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 Processor -- start 2024-09-21T01:22:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 -- start start 2024-09-21T01:22:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x56009910bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x560099115d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 0x560099116280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41b5960 con 0x560098d57520 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41b3200 con 0x560098d2e660 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3afc75740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41b1070 con 0x560098d36840 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x56009910bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 0x560099116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x56009910bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:60650/0 (socket says 172.21.15.39:60650) 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 0x560099116280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53896/0 (socket says 172.21.15.39:53896) 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.448+0000 7fe3a1e00640 1 -- 172.21.15.39:0/2576549884 learned_addr learned my addr 172.21.15.39:0/2576549884 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a1400640 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x560099115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a2800640 1 -- 172.21.15.39:0/2576549884 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 msgr2=0x560099115d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a2800640 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x560099115d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a2800640 1 -- 172.21.15.39:0/2576549884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 msgr2=0x56009910bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a2800640 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x56009910bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a2800640 1 -- 172.21.15.39:0/2576549884 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600991167c0 con 0x560098d57520 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe3a2800640 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 0x560099116280 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0ad8a0 tx=0x7fe39c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945dd40866a207b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe38ea00640 1 -- 172.21.15.39:0/2576549884 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0ad310 con 0x560098d57520 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.449+0000 7fe38ea00640 1 -- 172.21.15.39:0/2576549884 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe39c0ae2a0 con 0x560098d57520 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe38ea00640 1 -- 172.21.15.39:0/2576549884 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0b21c0 con 0x560098d57520 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 -- 172.21.15.39:0/2576549884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 msgr2=0x560099116280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 0x560099116280 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0ad8a0 tx=0x7fe39c004ab0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 -- 172.21.15.39:0/2576549884 shutdown_connections 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d57520 0x560099116280 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d36840 0x560099115d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 --2- 172.21.15.39:0/2576549884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x56009910bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 -- 172.21.15.39:0/2576549884 >> 172.21.15.39:0/2576549884 conn(0x560098d4b910 msgr2=0x5600990fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:14.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.450+0000 7fe3afc75740 1 -- 172.21.15.39:0/2576549884 shutdown_connections 2024-09-21T01:22:14.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.451+0000 7fe3afc75740 1 -- 172.21.15.39:0/2576549884 wait complete. 2024-09-21T01:22:14.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.451+0000 7fe3afc75740 1 -- start start 2024-09-21T01:22:14.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.451+0000 7fe3afc75740 1 Processor -- start 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x5600991122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 0x56009910cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d57520 0x56009910d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41b16c0 con 0x560098d36840 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41b2b30 con 0x560098d2e660 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a41b1d90 con 0x560098d57520 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3afc75740 10 client.0 _ll_register_callbacks cb 0x7fe39c0aa040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 0x56009910cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 0x56009910cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53898/0 (socket says 172.21.15.39:53898) 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 -- 172.21.15.39:0/1917456921 learned_addr learned my addr 172.21.15.39:0/1917456921 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a2800640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d57520 0x56009910d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x5600991122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 -- 172.21.15.39:0/1917456921 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d57520 msgr2=0x56009910d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d57520 0x56009910d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 -- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 msgr2=0x5600991122a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:14.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x5600991122a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.452+0000 7fe3a1400640 1 -- 172.21.15.39:0/1917456921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56009910d7e0 con 0x560098d36840 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe3a1400640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 0x56009910cd60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe390031440 tx=0x7fe39002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d89e5e85a833121 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe383e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe383e00640 1 -- 172.21.15.39:0/1917456921 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe390046070 con 0x560098d36840 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe383e00640 1 -- 172.21.15.39:0/1917456921 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9511+0+0 (secure 0 0 0) 0x7fe3900318b0 con 0x560098d36840 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe39c0ae2a0 con 0x560098d36840 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe383e00640 1 -- 172.21.15.39:0/1917456921 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fe390060050 con 0x560098d36840 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe3afc75740 10 client.16281 fetch_fsmap learned FSMap version 56 2024-09-21T01:22:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.453+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x560099187560 con 0x560098d36840 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe383e00640 1 -- 172.21.15.39:0/1917456921 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe39c0ae2a0 con 0x560098d36840 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe383e00640 1 -- 172.21.15.39:0/1917456921 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fe390055020 con 0x560098d36840 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe3afc75740 10 client.16281 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe3afc75740 10 client.16281 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe3afc75740 20 client.16281 populate_metadata read hostname 'smithi039' 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe3afc75740 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098ceb250 0x560099189d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe3afc75740 4 client.16281 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "eef6e4dc", "release": true, "await": true}] 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.454+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "eef6e4dc", "release": true, "await": true}) -- 0x560099072c90 con 0x560098ceb250 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.455+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098ceb250 0x560099189d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.455+0000 7fe3a1e00640 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098ceb250 0x560099189d50 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01276bf981db7c6 server_cookie=9eef8e0086ebfddd in_seq=0 out_seq=0 2024-09-21T01:22:14.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.455+0000 7fe383e00640 10 client.16281 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe383e00640 1 -- 172.21.15.39:0/1917456921 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560099072c90 con 0x560098ceb250 2024-09-21T01:22:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe383e00640 10 client.16281 handle_command_reply: tid=0 2024-09-21T01:22:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe3afc75740 1 client.16281 shutdown 2024-09-21T01:22:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 msgr2=0x56009910cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe3afc75740 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 0x56009910cd60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe390031440 tx=0x7fe39002a410 comp rx=0 tx=0).stop 2024-09-21T01:22:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 shutdown_connections 2024-09-21T01:22:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.496+0000 7fe3afc75740 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x560098ceb250 0x560099189d50 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x560098d57520 0x56009910d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560098d36840 0x56009910cd60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 1 --2- 172.21.15.39:0/1917456921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560098d2e660 0x5600991122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 >> 172.21.15.39:0/1917456921 conn(0x560098d4b910 msgr2=0x5600990fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe383e00640 0 client.16281 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 shutdown_connections 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 1 -- 172.21.15.39:0/1917456921 wait complete. 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.497+0000 7fe3afc75740 20 client.16281 trim_cache size 0 max 16384 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:22:14.499 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 56, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 72, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "eef6e4dc": { 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 72, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 130.2, 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:22:14.500 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:22:14.501 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:22:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3880b6680 msgr2=0x7fe3880b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3880b6680 0x7fe3880b8b90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe39002aa00 tx=0x7fe390041040 comp rx=0 tx=0).stop 2024-09-21T01:22:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 msgr2=0x7fe3a41a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 0x7fe3a41a1de0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00eb10 tx=0x7fe39c00c2d0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 shutdown_connections 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3a4109c40 0x7fe3a41a1de0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe3a41092f0 0x7fe3a419cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe3880b6680 0x7fe3880b8b90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 --2- 172.21.15.39:0/2306642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a41039b0 0x7fe3a419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 >> 172.21.15.39:0/2306642665 conn(0x7fe3a40ff400 msgr2=0x7fe3a41089b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.502+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 shutdown_connections 2024-09-21T01:22:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:14.503+0000 7fe3a3200640 1 -- 172.21.15.39:0/2306642665 wait complete. 2024-09-21T01:22:14.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76217 13.50 MB/sec execute 161 sec latency 8042.850 ms 2024-09-21T01:22:15.639 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'eef6e4dc', total seconds elapsed: 32.2 2024-09-21T01:22:15.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 76758 13.59 MB/sec execute 162 sec latency 160.169 ms 2024-09-21T01:22:16.152 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:15 smithi039 ceph-mon[30415]: pgmap v975: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 936 KiB/s wr, 107 op/s 2024-09-21T01:22:16.152 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:15 smithi039 ceph-mon[30415]: osdmap e115: 12 total, 12 up, 12 in 2024-09-21T01:22:16.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:15 smithi150 ceph-mon[34945]: pgmap v975: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 936 KiB/s wr, 107 op/s 2024-09-21T01:22:16.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:15 smithi150 ceph-mon[34945]: osdmap e115: 12 total, 12 up, 12 in 2024-09-21T01:22:16.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:15 smithi160 ceph-mon[34696]: pgmap v975: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 936 KiB/s wr, 107 op/s 2024-09-21T01:22:16.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:15 smithi160 ceph-mon[34696]: osdmap e115: 12 total, 12 up, 12 in 2024-09-21T01:22:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 77372 13.69 MB/sec execute 163 sec latency 156.250 ms 2024-09-21T01:22:17.776 INFO:tasks.workunit.client.0.smithi039.stdout: 10 77810 13.77 MB/sec execute 164 sec latency 489.466 ms 2024-09-21T01:22:18.146 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:17 smithi039 ceph-mon[30415]: pgmap v977: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 951 KiB/s wr, 115 op/s 2024-09-21T01:22:18.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:17 smithi160 ceph-mon[34696]: pgmap v977: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 951 KiB/s wr, 115 op/s 2024-09-21T01:22:18.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:17 smithi150 ceph-mon[34945]: pgmap v977: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 951 KiB/s wr, 115 op/s 2024-09-21T01:22:18.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 78169 13.76 MB/sec execute 165 sec latency 266.572 ms 2024-09-21T01:22:19.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 78892 13.92 MB/sec execute 166 sec latency 194.188 ms 2024-09-21T01:22:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:19 smithi039 ceph-mon[30415]: pgmap v978: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 171 op/s 2024-09-21T01:22:20.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:19 smithi150 ceph-mon[34945]: pgmap v978: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 171 op/s 2024-09-21T01:22:20.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:19 smithi160 ceph-mon[34696]: pgmap v978: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 171 op/s 2024-09-21T01:22:20.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 79667 14.12 MB/sec execute 167 sec latency 86.734 ms 2024-09-21T01:22:21.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 80352 14.28 MB/sec execute 168 sec latency 112.115 ms 2024-09-21T01:22:22.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:21 smithi039 ceph-mon[30415]: pgmap v979: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-09-21T01:22:22.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:21 smithi150 ceph-mon[34945]: pgmap v979: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-09-21T01:22:22.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:21 smithi160 ceph-mon[34696]: pgmap v979: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-09-21T01:22:22.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 80826 14.35 MB/sec execute 169 sec latency 253.295 ms 2024-09-21T01:22:23.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 81296 14.38 MB/sec execute 170 sec latency 226.666 ms 2024-09-21T01:22:24.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:23 smithi150 ceph-mon[34945]: pgmap v980: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.6 MiB/s wr, 305 op/s 2024-09-21T01:22:24.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:23 smithi160 ceph-mon[34696]: pgmap v980: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.6 MiB/s wr, 305 op/s 2024-09-21T01:22:24.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:23 smithi039 ceph-mon[30415]: pgmap v980: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.6 MiB/s wr, 305 op/s 2024-09-21T01:22:24.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 81925 14.48 MB/sec execute 171 sec latency 158.327 ms 2024-09-21T01:22:25.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 82673 14.62 MB/sec execute 172 sec latency 102.675 ms 2024-09-21T01:22:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:25 smithi160 ceph-mon[34696]: pgmap v981: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-09-21T01:22:25.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:25 smithi160 ceph-mon[34696]: osdmap e116: 12 total, 12 up, 12 in 2024-09-21T01:22:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:25 smithi039 ceph-mon[30415]: pgmap v981: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-09-21T01:22:25.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:25 smithi039 ceph-mon[30415]: osdmap e116: 12 total, 12 up, 12 in 2024-09-21T01:22:26.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:25 smithi150 ceph-mon[34945]: pgmap v981: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-09-21T01:22:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:25 smithi150 ceph-mon[34945]: osdmap e116: 12 total, 12 up, 12 in 2024-09-21T01:22:26.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 83379 14.79 MB/sec execute 173 sec latency 107.615 ms 2024-09-21T01:22:26.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:26 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:22:27.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:26 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:22:27.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:26 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:22:27.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 83989 14.89 MB/sec execute 174 sec latency 138.541 ms 2024-09-21T01:22:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:27 smithi039 ceph-mon[30415]: pgmap v983: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 434 op/s 2024-09-21T01:22:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:27 smithi150 ceph-mon[34945]: pgmap v983: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 434 op/s 2024-09-21T01:22:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:27 smithi160 ceph-mon[34696]: pgmap v983: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 434 op/s 2024-09-21T01:22:28.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 84390 14.90 MB/sec execute 175 sec latency 266.289 ms 2024-09-21T01:22:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:22:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:22:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:22:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:22:29.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:29.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:22:29.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:22:29.754 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:29 smithi039 ceph-mon[30415]: pgmap v984: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.4 MiB/s wr, 477 op/s 2024-09-21T01:22:29.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 84871 14.99 MB/sec execute 176 sec latency 229.901 ms 2024-09-21T01:22:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:29 smithi150 ceph-mon[34945]: pgmap v984: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.4 MiB/s wr, 477 op/s 2024-09-21T01:22:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:29 smithi160 ceph-mon[34696]: pgmap v984: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.4 MiB/s wr, 477 op/s 2024-09-21T01:22:30.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 85462 15.09 MB/sec execute 177 sec latency 273.959 ms 2024-09-21T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:31 smithi039 ceph-mon[30415]: pgmap v985: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 473 op/s 2024-09-21T01:22:31.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 86091 15.16 MB/sec execute 178 sec latency 95.488 ms 2024-09-21T01:22:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:31 smithi150 ceph-mon[34945]: pgmap v985: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 473 op/s 2024-09-21T01:22:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:31 smithi160 ceph-mon[34696]: pgmap v985: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 473 op/s 2024-09-21T01:22:32.777 INFO:tasks.workunit.client.0.smithi039.stdout: 10 86800 15.34 MB/sec execute 179 sec latency 136.156 ms 2024-09-21T01:22:33.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:33 smithi039 ceph-mon[30415]: pgmap v986: 129 pgs: 129 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 432 op/s 2024-09-21T01:22:33.778 INFO:tasks.workunit.client.0.smithi039.stdout: 10 87340 15.42 MB/sec execute 180 sec latency 158.574 ms 2024-09-21T01:22:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:33 smithi150 ceph-mon[34945]: pgmap v986: 129 pgs: 129 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 432 op/s 2024-09-21T01:22:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:33 smithi160 ceph-mon[34696]: pgmap v986: 129 pgs: 129 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 432 op/s 2024-09-21T01:22:34.583 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:22:34.583 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.15 2024-09-21T01:22:34.583 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:22:34.778 INFO:tasks.workunit.client.0.smithi039.stdout: 10 87870 15.44 MB/sec execute 181 sec latency 228.534 ms 2024-09-21T01:22:35.256 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:22:35.778 INFO:tasks.workunit.client.0.smithi039.stdout: 10 88337 15.53 MB/sec execute 182 sec latency 266.113 ms 2024-09-21T01:22:35.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:35 smithi039 ceph-mon[30415]: pgmap v987: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 407 op/s 2024-09-21T01:22:36.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:35 smithi150 ceph-mon[34945]: pgmap v987: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 407 op/s 2024-09-21T01:22:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:35 smithi160 ceph-mon[34696]: pgmap v987: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 407 op/s 2024-09-21T01:22:36.778 INFO:tasks.workunit.client.0.smithi039.stdout: 10 88817 15.58 MB/sec execute 183 sec latency 411.904 ms 2024-09-21T01:22:37.778 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89166 15.57 MB/sec execute 184 sec latency 485.151 ms 2024-09-21T01:22:37.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:37 smithi039 ceph-mon[30415]: pgmap v988: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 431 op/s 2024-09-21T01:22:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:37 smithi150 ceph-mon[34945]: pgmap v988: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 431 op/s 2024-09-21T01:22:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:37 smithi160 ceph-mon[34696]: pgmap v988: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 431 op/s 2024-09-21T01:22:38.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 -- 172.21.15.39:0/1780993052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10fd10 msgr2=0x7f9abc110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:38.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 --2- 172.21.15.39:0/1780993052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10fd10 0x7f9abc110120 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009860 tx=0x7f9aa802a380 comp rx=0 tx=0).stop 2024-09-21T01:22:38.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 -- 172.21.15.39:0/1780993052 shutdown_connections 2024-09-21T01:22:38.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 --2- 172.21.15.39:0/1780993052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 0x7f9abc110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:38.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 --2- 172.21.15.39:0/1780993052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10fd10 0x7f9abc110120 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009860 tx=0x7f9aa802a380 comp rx=0 tx=0).stop 2024-09-21T01:22:38.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 --2- 172.21.15.39:0/1780993052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc10a650 0x7f9abc10aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:38.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.638+0000 7f9ac2800640 1 -- 172.21.15.39:0/1780993052 >> 172.21.15.39:0/1780993052 conn(0x7f9abc06e9f0 msgr2=0x7f9abc06ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:38.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.645+0000 7f9ac2800640 1 -- 172.21.15.39:0/1780993052 shutdown_connections 2024-09-21T01:22:38.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.646+0000 7f9ac2800640 1 -- 172.21.15.39:0/1780993052 wait complete. 2024-09-21T01:22:38.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.651+0000 7f9ac2800640 1 Processor -- start 2024-09-21T01:22:38.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.651+0000 7f9ac2800640 1 -- start start 2024-09-21T01:22:38.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.657+0000 7f9ac2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 0x7f9abc11e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:38.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.657+0000 7f9ac2800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10a650 0x7f9abc11baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:38.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.657+0000 7f9ac1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 0x7f9abc11e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:38.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 0x7f9abc11e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56584/0 (socket says 172.21.15.39:56584) 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac2800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc11c030 0x7f9abc11c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10a650 0x7f9abc11baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10a650 0x7f9abc11baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:33990/0 (socket says 172.21.15.39:33990) 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac1400640 1 -- 172.21.15.39:0/130365964 learned_addr learned my addr 172.21.15.39:0/130365964 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc119c80 con 0x7f9abc074c80 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc119b00 con 0x7f9abc10a650 2024-09-21T01:22:38.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc119e00 con 0x7f9abc11c030 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.658+0000 7f9ac1e00640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc11c030 0x7f9abc11c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.659+0000 7f9ac1e00640 1 -- 172.21.15.39:0/130365964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10a650 msgr2=0x7f9abc11baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.659+0000 7f9ac1e00640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10a650 0x7f9abc11baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.659+0000 7f9ac1e00640 1 -- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 msgr2=0x7f9abc11e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.659+0000 7f9ac1e00640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 0x7f9abc11e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.659+0000 7f9ac1e00640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc1b5790 con 0x7f9abc11c030 2024-09-21T01:22:38.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.660+0000 7f9ac1e00640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc11c030 0x7f9abc11c440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400e940 tx=0x7f9ab400ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:38.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.661+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4004d20 con 0x7f9abc11c030 2024-09-21T01:22:38.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.661+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ab4005020 con 0x7f9abc11c030 2024-09-21T01:22:38.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.661+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40116d0 con 0x7f9abc11c030 2024-09-21T01:22:38.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.661+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9abc1b5a80 con 0x7f9abc11c030 2024-09-21T01:22:38.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.661+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9abc1b5f40 con 0x7f9abc11c030 2024-09-21T01:22:38.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.664+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9abc10b650 con 0x7f9abc11c030 2024-09-21T01:22:38.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.668+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f9ab4005460 con 0x7f9abc11c030 2024-09-21T01:22:38.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.674+0000 7f9aba000640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9aa40b6800 0x7f9aa40b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:38.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.675+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9511+0+0 (secure 0 0 0) 0x7f9ab40e7ae0 con 0x7f9abc11c030 2024-09-21T01:22:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.676+0000 7f9ac1400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9aa40b6800 0x7f9aa40b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.678+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9ab4016030 con 0x7f9abc11c030 2024-09-21T01:22:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:38.678+0000 7f9ac1400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9aa40b6800 0x7f9aa40b8d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9abc11e550 tx=0x7f9ab0009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:38.796 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89644 15.61 MB/sec execute 185 sec latency 325.735 ms 2024-09-21T01:22:39.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.235+0000 7f9ac2800640 1 -- 172.21.15.39:0/130365964 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9abc1b62f0 con 0x7f9abc11c030 2024-09-21T01:22:39.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.237+0000 7f9aba000640 1 -- 172.21.15.39:0/130365964 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11912 (secure 0 0 0) 0x7f9ab40adde0 con 0x7f9abc11c030 2024-09-21T01:22:39.242 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:22:39.242 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":56,"btime":"2024-09-21T01:21:44:738665+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:44.386909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:22:39.243 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 56 2024-09-21T01:22:39.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 -- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9aa40b6800 msgr2=0x7f9aa40b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:39.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9aa40b6800 0x7f9aa40b8d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9abc11e550 tx=0x7f9ab0009260 comp rx=0 tx=0).stop 2024-09-21T01:22:39.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 -- 172.21.15.39:0/130365964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc11c030 msgr2=0x7f9abc11c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:39.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc11c030 0x7f9abc11c440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400e940 tx=0x7f9ab400ee00 comp rx=0 tx=0).stop 2024-09-21T01:22:39.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 -- 172.21.15.39:0/130365964 shutdown_connections 2024-09-21T01:22:39.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9abc11c030 0x7f9abc11c440 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:39.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f9aa40b6800 0x7f9aa40b8d10 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9abc11e550 tx=0x7f9ab0009260 comp rx=0 tx=0).stop 2024-09-21T01:22:39.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc10a650 0x7f9abc11baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:39.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 --2- 172.21.15.39:0/130365964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9abc074c80 0x7f9abc11e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:39.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.245+0000 7f9aaf400640 1 -- 172.21.15.39:0/130365964 >> 172.21.15.39:0/130365964 conn(0x7f9abc06e9f0 msgr2=0x7f9abc115de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:39.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.248+0000 7f9aaf400640 1 -- 172.21.15.39:0/130365964 shutdown_connections 2024-09-21T01:22:39.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:39.248+0000 7f9aaf400640 1 -- 172.21.15.39:0/130365964 wait complete. 2024-09-21T01:22:39.755 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:39 smithi039 ceph-mon[30415]: pgmap v989: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.0 MiB/s wr, 471 op/s 2024-09-21T01:22:39.755 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:39 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/130365964' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:22:39.800 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89649 15.53 MB/sec execute 186 sec latency 1332.115 ms 2024-09-21T01:22:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:39 smithi150 ceph-mon[34945]: pgmap v989: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.0 MiB/s wr, 471 op/s 2024-09-21T01:22:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:39 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/130365964' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:22:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:39 smithi160 ceph-mon[34696]: pgmap v989: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.0 MiB/s wr, 471 op/s 2024-09-21T01:22:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:39 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/130365964' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:22:40.448 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0' 2024-09-21T01:22:40.448 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0 2024-09-21T01:22:40.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89649 15.45 MB/sec execute 187 sec latency 2332.670 ms 2024-09-21T01:22:40.859 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:22:41.707 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:41 smithi039 ceph-mon[30415]: pgmap v990: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.2 MiB/s wr, 370 op/s 2024-09-21T01:22:41.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89650 15.37 MB/sec execute 188 sec latency 3286.703 ms 2024-09-21T01:22:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:41 smithi150 ceph-mon[34945]: pgmap v990: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.2 MiB/s wr, 370 op/s 2024-09-21T01:22:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:41 smithi160 ceph-mon[34696]: pgmap v990: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.2 MiB/s wr, 370 op/s 2024-09-21T01:22:42.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89790 15.33 MB/sec execute 189 sec latency 4286.746 ms 2024-09-21T01:22:43.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 -- 172.21.15.39:0/3368612751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864075730 msgr2=0x7f6864075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:43.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 --2- 172.21.15.39:0/3368612751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864075730 0x7f6864075b40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6854009860 tx=0x7f685402a340 comp rx=0 tx=0).stop 2024-09-21T01:22:43.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 -- 172.21.15.39:0/3368612751 shutdown_connections 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 --2- 172.21.15.39:0/3368612751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f686410ffd0 0x7f68641123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 --2- 172.21.15.39:0/3368612751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6864076080 0x7f686410fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 --2- 172.21.15.39:0/3368612751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864075730 0x7f6864075b40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 -- 172.21.15.39:0/3368612751 >> 172.21.15.39:0/3368612751 conn(0x7f68640ff480 msgr2=0x7f68641018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 -- 172.21.15.39:0/3368612751 shutdown_connections 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.757+0000 7f6863200640 1 -- 172.21.15.39:0/3368612751 wait complete. 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.758+0000 7f6863200640 1 Processor -- start 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.758+0000 7f6863200640 1 -- start start 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6863200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 0x7f686419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6863200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864076080 0x7f686419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6863200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f686410ffd0 0x7f68641a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6863200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6864115b10 con 0x7f686410ffd0 2024-09-21T01:22:43.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6863200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6864115990 con 0x7f6864075730 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6863200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6864115c90 con 0x7f6864076080 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6861e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 0x7f686419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6862800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f686410ffd0 0x7f68641a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6861e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 0x7f686419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:34002/0 (socket says 172.21.15.39:34002) 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6861e00640 1 -- 172.21.15.39:0/1675928423 learned_addr learned my addr 172.21.15.39:0/1675928423 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6862800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f686410ffd0 0x7f68641a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56604/0 (socket says 172.21.15.39:56604) 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.759+0000 7f6861400640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864076080 0x7f686419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861e00640 1 -- 172.21.15.39:0/1675928423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864076080 msgr2=0x7f686419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861e00640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864076080 0x7f686419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861e00640 1 -- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f686410ffd0 msgr2=0x7f68641a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861e00640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f686410ffd0 0x7f68641a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861e00640 1 -- 172.21.15.39:0/1675928423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68641a24d0 con 0x7f6864075730 2024-09-21T01:22:43.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861400640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864076080 0x7f686419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6861e00640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 0x7f686419c560 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f68540303f0 tx=0x7f6854032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f685aa00640 1 -- 172.21.15.39:0/1675928423 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854040020 con 0x7f6864075730 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f685aa00640 1 -- 172.21.15.39:0/1675928423 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6854030dd0 con 0x7f6864075730 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68641a2760 con 0x7f6864075730 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.760+0000 7f685aa00640 1 -- 172.21.15.39:0/1675928423 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854037040 con 0x7f6864075730 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.761+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68641a8f20 con 0x7f6864075730 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.763+0000 7f686fa73740 1 Processor -- start 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 -- start start 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 0x55d9e6fcbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fcc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c03b70 0x55d9e6fcc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6f41a90 con 0x55d9e6c03b70 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6f30ec0 con 0x55d9e6bfb990 2024-09-21T01:22:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f686fa73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6f34cd0 con 0x55d9e6c24850 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f6862800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 0x55d9e6fcbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f6862800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 0x55d9e6fcbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34386/0 (socket says 172.21.15.39:34386) 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f6862800640 1 -- 172.21.15.39:0/4061576099 learned_addr learned my addr 172.21.15.39:0/4061576099 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.764+0000 7f6861400640 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c03b70 0x55d9e6fcc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f685aa00640 1 -- 172.21.15.39:0/1675928423 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6854043020 con 0x7f6864075730 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6861e00640 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fcc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6862800640 1 -- 172.21.15.39:0/4061576099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 msgr2=0x55d9e6fcc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6862800640 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fcc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6862800640 1 -- 172.21.15.39:0/4061576099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c03b70 msgr2=0x55d9e6fcc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6862800640 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c03b70 0x55d9e6fcc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6862800640 1 -- 172.21.15.39:0/4061576099 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9e6ed69d0 con 0x55d9e6c24850 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.765+0000 7f6862800640 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 0x55d9e6fcbb90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f685c009fd0 tx=0x7f685c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43924a6b645f8fd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.766+0000 7f684f400640 1 -- 172.21.15.39:0/4061576099 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c004d20 con 0x55d9e6c24850 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.766+0000 7f684f400640 1 -- 172.21.15.39:0/4061576099 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685c004050 con 0x55d9e6c24850 2024-09-21T01:22:43.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.766+0000 7f684f400640 1 -- 172.21.15.39:0/4061576099 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c005280 con 0x55d9e6c24850 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.766+0000 7f685aa00640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68480b6770 0x7f68480b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f685aa00640 1 -- 172.21.15.39:0/1675928423 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9511+0+0 (secure 0 0 0) 0x7f6854107670 con 0x7f6864075730 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f6861e00640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68480b6770 0x7f68480b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 -- 172.21.15.39:0/4061576099 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 msgr2=0x55d9e6fcbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 0x55d9e6fcbb90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f685c009fd0 tx=0x7f685c0118d0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 -- 172.21.15.39:0/4061576099 shutdown_connections 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c03b70 0x55d9e6fcc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fcc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 --2- 172.21.15.39:0/4061576099 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c24850 0x55d9e6fcbb90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 -- 172.21.15.39:0/4061576099 >> 172.21.15.39:0/4061576099 conn(0x55d9e6c18c40 msgr2=0x55d9e6fc9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 -- 172.21.15.39:0/4061576099 shutdown_connections 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.767+0000 7f686fa73740 1 -- 172.21.15.39:0/4061576099 wait complete. 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.768+0000 7f6861e00640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68480b6770 0x7f68480b8c80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55d9e6fca810 tx=0x7f6854004610 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:43.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.768+0000 7f686fa73740 1 -- start start 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.768+0000 7f686fa73740 1 Processor -- start 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fd7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6d927b0 con 0x55d9e6c24850 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6ed5650 con 0x55d9e6c03b70 2024-09-21T01:22:43.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6fda370 con 0x55d9e6bfb990 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f686fa73740 10 client.0 _ll_register_callbacks cb 0x55d9e6c3c0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f6862800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f6862800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34392/0 (socket says 172.21.15.39:34392) 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f6862800640 1 -- 172.21.15.39:0/1749889974 learned_addr learned my addr 172.21.15.39:0/1749889974 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f6861400640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fd7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.769+0000 7f6862800640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6862800640 1 -- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 msgr2=0x55d9e6fd7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6862800640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fd7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6862800640 1 -- 172.21.15.39:0/1749889974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 msgr2=0x55d9e6fe3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6862800640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6862800640 1 -- 172.21.15.39:0/1749889974 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9e6c9eba0 con 0x55d9e6bfb990 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6861400640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fd7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6862800640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd7670 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f685c004930 tx=0x7f685c009be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20fba5524e5764d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:43.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6843e00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.770+0000 7f6843e00640 1 -- 172.21.15.39:0/1749889974 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c005ad0 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f685c004050 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f6843e00640 1 -- 172.21.15.39:0/1749889974 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9511+0+0 (secure 0 0 0) 0x7f685c01f040 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f6843e00640 1 -- 172.21.15.39:0/1749889974 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f685c034050 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 10 client.26302 fetch_fsmap learned FSMap version 56 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55d9e6fe62e0 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f6843e00640 1 -- 172.21.15.39:0/1749889974 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685c004050 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f6843e00640 1 -- 172.21.15.39:0/1749889974 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f685c0056b0 con 0x55d9e6bfb990 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 10 client.26302 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 10 client.26302 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 20 client.26302 populate_metadata read hostname 'smithi039' 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6be31d0 0x55d9e6fe96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 4 client.26302 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.771+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9e6f439e0 con 0x55d9e6be31d0 2024-09-21T01:22:43.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.772+0000 7f6861400640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6be31d0 0x55d9e6fe96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.772+0000 7f6861400640 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6be31d0 0x55d9e6fe96a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7616a34db81ddfec server_cookie=29b4d4cb172a11c2 in_seq=0 out_seq=0 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.772+0000 7f6843e00640 10 client.26302 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f6843e00640 1 -- 172.21.15.39:0/1749889974 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d9e6f439e0 con 0x55d9e6be31d0 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f6843e00640 10 client.26302 handle_command_reply: tid=0 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 client.26302 shutdown 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 msgr2=0x55d9e6fd7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd7670 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f685c004930 tx=0x7f685c009be0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 shutdown_connections 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6be31d0 0x55d9e6fe96a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fd7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 --2- 172.21.15.39:0/1749889974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd7670 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f6843e00640 0 client.26302 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.773+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 >> 172.21.15.39:0/1749889974 conn(0x55d9e6c18c40 msgr2=0x55d9e6fced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.774+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 shutdown_connections 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.774+0000 7f686fa73740 1 -- 172.21.15.39:0/1749889974 wait complete. 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.774+0000 7f686fa73740 20 client.26302 trim_cache size 0 max 16384 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.793+0000 7f686fa73740 1 Processor -- start 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.793+0000 7f686fa73740 1 -- start start 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f686fa73740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f686fa73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 0x55d9e6fe3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f686fa73740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f686fa73740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f68641b59b0 con 0x55d9e6c24850 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f686fa73740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68641b3250 con 0x55d9e6bfb990 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f686fa73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f68641b10c0 con 0x55d9e6c03b70 2024-09-21T01:22:43.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f6861400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 0x55d9e6fe3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f6862800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f6861400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 0x55d9e6fe3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:34402/0 (socket says 172.21.15.39:34402) 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f6861400640 1 -- 172.21.15.39:0/737257665 learned_addr learned my addr 172.21.15.39:0/737257665 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.794+0000 7f6862800640 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f6861400640 1 -- 172.21.15.39:0/737257665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 msgr2=0x55d9e6fd9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f6861400640 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f6861400640 1 -- 172.21.15.39:0/737257665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 msgr2=0x55d9e6fe3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f6861400640 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f6861400640 1 -- 172.21.15.39:0/737257665 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9e6fc5130 con 0x55d9e6c03b70 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f6861400640 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 0x55d9e6fe3590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f686406b920 tx=0x7f6850009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4806a8403a09c6d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f684ea00640 1 -- 172.21.15.39:0/737257665 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685000fb30 con 0x55d9e6c03b70 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f684ea00640 1 -- 172.21.15.39:0/737257665 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685000fcd0 con 0x55d9e6c03b70 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.795+0000 7f684ea00640 1 -- 172.21.15.39:0/737257665 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850003d50 con 0x55d9e6c03b70 2024-09-21T01:22:43.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 -- 172.21.15.39:0/737257665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 msgr2=0x55d9e6fe3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 0x55d9e6fe3590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f686406b920 tx=0x7f6850009260 comp rx=0 tx=0).stop 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 -- 172.21.15.39:0/737257665 shutdown_connections 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fe3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6c03b70 0x55d9e6fe3590 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 --2- 172.21.15.39:0/737257665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6bfb990 0x55d9e6fd9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 -- 172.21.15.39:0/737257665 >> 172.21.15.39:0/737257665 conn(0x55d9e6c18c40 msgr2=0x55d9e6fc8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 -- 172.21.15.39:0/737257665 shutdown_connections 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.796+0000 7f686fa73740 1 -- 172.21.15.39:0/737257665 wait complete. 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.797+0000 7f686fa73740 1 -- start start 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.797+0000 7f686fa73740 1 Processor -- start 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.797+0000 7f686fa73740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fdb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.797+0000 7f686fa73740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fdbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f686fa73740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fdc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f686fa73740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f68641b1710 con 0x55d9e6c24850 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f686fa73740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68641b2b80 con 0x55d9e6c03b70 2024-09-21T01:22:43.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f686fa73740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f68641b1de0 con 0x55d9e6bfb990 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f686fa73740 10 client.0 _ll_register_callbacks cb 0x7f685c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fdbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fdbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:34034/0 (socket says 172.21.15.39:34034) 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 -- 172.21.15.39:0/2606713627 learned_addr learned my addr 172.21.15.39:0/2606713627 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6862800640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fdb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6862800640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fdc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 -- 172.21.15.39:0/2606713627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 msgr2=0x55d9e6fdb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fdb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 -- 172.21.15.39:0/2606713627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 msgr2=0x55d9e6fdc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fdc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.798+0000 7f6861400640 1 -- 172.21.15.39:0/2606713627 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9e6fc4f10 con 0x55d9e6c03b70 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f6861400640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fdbaf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6850013010 tx=0x7f685000fbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd1f6180f9faa598 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f6843e00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f6843e00640 1 -- 172.21.15.39:0/2606713627 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850032020 con 0x55d9e6c03b70 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f685000fcd0 con 0x55d9e6c03b70 2024-09-21T01:22:43.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f6843e00640 1 -- 172.21.15.39:0/2606713627 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9511+0+0 (secure 0 0 0) 0x7f6850003bd0 con 0x55d9e6c03b70 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f6843e00640 1 -- 172.21.15.39:0/2606713627 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f6850003dd0 con 0x55d9e6c03b70 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f686fa73740 10 client.26324 fetch_fsmap learned FSMap version 56 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.799+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55d9e7053f10 con 0x55d9e6c03b70 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f6843e00640 1 -- 172.21.15.39:0/2606713627 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685000fcd0 con 0x55d9e6c03b70 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f6843e00640 1 -- 172.21.15.39:0/2606713627 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f6850004a70 con 0x55d9e6c03b70 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f686fa73740 10 client.26324 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f686fa73740 10 client.26324 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f686fa73740 20 client.26324 populate_metadata read hostname 'smithi039' 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f686fa73740 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6bb8580 0x55d9e70566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f686fa73740 4 client.26324 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}] 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}) -- 0x55d9e6f3f490 con 0x55d9e6bb8580 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.800+0000 7f6862800640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6bb8580 0x55d9e70566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.801+0000 7f6862800640 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6bb8580 0x55d9e70566e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5459c91353da7742 server_cookie=9698e95a479f29ae in_seq=0 out_seq=0 2024-09-21T01:22:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:22:42.801+0000 7f6843e00640 10 client.26324 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:22:43.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89795 15.25 MB/sec execute 190 sec latency 5286.790 ms 2024-09-21T01:22:43.893 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:43 smithi039 ceph-mon[30415]: pgmap v991: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.1 MiB/s wr, 375 op/s 2024-09-21T01:22:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:43 smithi150 ceph-mon[34945]: pgmap v991: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.1 MiB/s wr, 375 op/s 2024-09-21T01:22:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:43 smithi160 ceph-mon[34696]: pgmap v991: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.1 MiB/s wr, 375 op/s 2024-09-21T01:22:44.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89796 15.17 MB/sec execute 191 sec latency 6286.849 ms 2024-09-21T01:22:45.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 15.09 MB/sec execute 192 sec latency 7286.907 ms 2024-09-21T01:22:45.873 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:45 smithi039 ceph-mon[30415]: pgmap v992: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 8.9 MiB/s wr, 378 op/s 2024-09-21T01:22:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:45 smithi150 ceph-mon[34945]: pgmap v992: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 8.9 MiB/s wr, 378 op/s 2024-09-21T01:22:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:45 smithi160 ceph-mon[34696]: pgmap v992: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 8.9 MiB/s wr, 378 op/s 2024-09-21T01:22:46.801 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 15.01 MB/sec execute 193 sec latency 8286.961 ms 2024-09-21T01:22:47.815 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 14.93 MB/sec execute 194 sec latency 9301.213 ms 2024-09-21T01:22:47.834 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:47 smithi039 ceph-mon[30415]: pgmap v993: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 350 op/s 2024-09-21T01:22:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:47 smithi150 ceph-mon[34945]: pgmap v993: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 350 op/s 2024-09-21T01:22:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:47 smithi160 ceph-mon[34696]: pgmap v993: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 350 op/s 2024-09-21T01:22:48.815 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 14.86 MB/sec execute 195 sec latency 10301.294 ms 2024-09-21T01:22:49.815 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 14.78 MB/sec execute 196 sec latency 11301.360 ms 2024-09-21T01:22:49.826 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:49 smithi039 ceph-mon[30415]: pgmap v994: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 12 MiB/s wr, 418 op/s 2024-09-21T01:22:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:49 smithi150 ceph-mon[34945]: pgmap v994: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 12 MiB/s wr, 418 op/s 2024-09-21T01:22:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:49 smithi160 ceph-mon[34696]: pgmap v994: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 12 MiB/s wr, 418 op/s 2024-09-21T01:22:50.815 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 14.70 MB/sec execute 197 sec latency 12301.432 ms 2024-09-21T01:22:51.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:51 smithi039 ceph-mon[30415]: pgmap v995: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 357 op/s 2024-09-21T01:22:51.815 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89797 14.63 MB/sec execute 198 sec latency 13301.497 ms 2024-09-21T01:22:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:51 smithi150 ceph-mon[34945]: pgmap v995: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 357 op/s 2024-09-21T01:22:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:51 smithi160 ceph-mon[34696]: pgmap v995: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 357 op/s 2024-09-21T01:22:52.815 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89800 14.56 MB/sec execute 199 sec latency 14015.534 ms 2024-09-21T01:22:53.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.48 MB/sec execute 200 sec latency 11884.881 ms 2024-09-21T01:22:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:53 smithi150 ceph-mon[34945]: pgmap v996: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 11 MiB/s wr, 359 op/s 2024-09-21T01:22:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:53 smithi160 ceph-mon[34696]: pgmap v996: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 11 MiB/s wr, 359 op/s 2024-09-21T01:22:54.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:53 smithi039 ceph-mon[30415]: pgmap v996: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 11 MiB/s wr, 359 op/s 2024-09-21T01:22:54.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.41 MB/sec execute 201 sec latency 12884.945 ms 2024-09-21T01:22:55.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.34 MB/sec execute 202 sec latency 13884.993 ms 2024-09-21T01:22:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:55 smithi150 ceph-mon[34945]: pgmap v997: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.5 MiB/s wr, 346 op/s 2024-09-21T01:22:56.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:55 smithi160 ceph-mon[34696]: pgmap v997: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.5 MiB/s wr, 346 op/s 2024-09-21T01:22:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:55 smithi039 ceph-mon[30415]: pgmap v997: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.5 MiB/s wr, 346 op/s 2024-09-21T01:22:56.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.27 MB/sec execute 203 sec latency 14885.039 ms 2024-09-21T01:22:57.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.20 MB/sec execute 204 sec latency 15885.086 ms 2024-09-21T01:22:58.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:57 smithi150 ceph-mon[34945]: pgmap v998: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.7 MiB/s wr, 310 op/s 2024-09-21T01:22:58.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:57 smithi160 ceph-mon[34696]: pgmap v998: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.7 MiB/s wr, 310 op/s 2024-09-21T01:22:58.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:57 smithi039 ceph-mon[30415]: pgmap v998: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.7 MiB/s wr, 310 op/s 2024-09-21T01:22:58.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.13 MB/sec execute 205 sec latency 16885.132 ms 2024-09-21T01:22:59.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89801 14.06 MB/sec execute 206 sec latency 17885.180 ms 2024-09-21T01:23:00.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:22:59 smithi150 ceph-mon[34945]: pgmap v999: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 276 op/s 2024-09-21T01:23:00.095 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:22:59 smithi039 ceph-mon[30415]: pgmap v999: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 276 op/s 2024-09-21T01:23:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:22:59 smithi160 ceph-mon[34696]: pgmap v999: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 276 op/s 2024-09-21T01:23:00.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f6843e00640 1 -- 172.21.15.39:0/2606713627 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d9e6f3f490 con 0x55d9e6bb8580 2024-09-21T01:23:00.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f6843e00640 10 client.26324 handle_command_reply: tid=0 2024-09-21T01:23:00.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 client.26324 shutdown 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 msgr2=0x55d9e6fdbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fdbaf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6850013010 tx=0x7f685000fbc0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 shutdown_connections 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55d9e6bb8580 0x55d9e70566e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d9e6c24850 0x55d9e6fdc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9e6c03b70 0x55d9e6fdbaf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 --2- 172.21.15.39:0/2606713627 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55d9e6bfb990 0x55d9e6fdb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 >> 172.21.15.39:0/2606713627 conn(0x55d9e6c18c40 msgr2=0x55d9e6fd8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.498+0000 7f6843e00640 0 client.26324 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.499+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 shutdown_connections 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.499+0000 7f686fa73740 1 -- 172.21.15.39:0/2606713627 wait complete. 2024-09-21T01:23:00.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.499+0000 7f686fa73740 20 client.26324 trim_cache size 0 max 16384 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 56, 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 74, 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "edd8abad": { 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 74, 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:00.538 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 142.2, 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:00.539 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.500+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68480b6770 msgr2=0x7f68480b8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.500+0000 7f6863200640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68480b6770 0x7f68480b8c80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55d9e6fca810 tx=0x7f6854004610 comp rx=0 tx=0).stop 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.500+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 msgr2=0x7f686419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.500+0000 7f6863200640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 0x7f686419c560 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f68540303f0 tx=0x7f6854032da0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 shutdown_connections 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f686410ffd0 0x7f68641a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6864076080 0x7f686419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68480b6770 0x7f68480b8c80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 --2- 172.21.15.39:0/1675928423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6864075730 0x7f686419c560 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:00.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 >> 172.21.15.39:0/1675928423 conn(0x7f68640ff480 msgr2=0x7f6864109b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:00.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 shutdown_connections 2024-09-21T01:23:00.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:00.501+0000 7f6863200640 1 -- 172.21.15.39:0/1675928423 wait complete. 2024-09-21T01:23:00.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89808 13.99 MB/sec execute 207 sec latency 18575.345 ms 2024-09-21T01:23:01.600 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'edd8abad', quiesce took 27.0 seconds. Will release after: -4.850000000000001 2024-09-21T01:23:01.601 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:23:01.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89808 13.93 MB/sec execute 208 sec latency 9271.935 ms 2024-09-21T01:23:01.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:01 smithi039 ceph-mon[30415]: pgmap v1000: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T01:23:01.982 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:23:02.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:01 smithi150 ceph-mon[34945]: pgmap v1000: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T01:23:02.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:01 smithi160 ceph-mon[34696]: pgmap v1000: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-09-21T01:23:02.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.737+0000 7fe50b200640 1 -- 172.21.15.39:0/725983986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c10ff10 msgr2=0x7fe50c112350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:02.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.737+0000 7fe50b200640 1 --2- 172.21.15.39:0/725983986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c10ff10 0x7fe50c112350 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe504009ca0 tx=0x7fe50402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 -- 172.21.15.39:0/725983986 shutdown_connections 2024-09-21T01:23:02.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 --2- 172.21.15.39:0/725983986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c10ff10 0x7fe50c112350 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 --2- 172.21.15.39:0/725983986 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c076100 0x7fe50c10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 --2- 172.21.15.39:0/725983986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c0757b0 0x7fe50c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 -- 172.21.15.39:0/725983986 >> 172.21.15.39:0/725983986 conn(0x7fe50c0ff440 msgr2=0x7fe50c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:02.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 -- 172.21.15.39:0/725983986 shutdown_connections 2024-09-21T01:23:02.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.738+0000 7fe50b200640 1 -- 172.21.15.39:0/725983986 wait complete. 2024-09-21T01:23:02.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 Processor -- start 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 -- start start 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c0757b0 0x7fe50c19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 0x7fe50c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c10ff10 0x7fe50c1a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c115b10 con 0x7fe50c076100 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c115990 con 0x7fe50c10ff10 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.739+0000 7fe50b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c115c90 con 0x7fe50c0757b0 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 0x7fe50c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 0x7fe50c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34106/0 (socket says 172.21.15.39:34106) 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 -- 172.21.15.39:0/1920621706 learned_addr learned my addr 172.21.15.39:0/1920621706 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe509e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c0757b0 0x7fe50c19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:02.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe50a800640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c10ff10 0x7fe50c1a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 -- 172.21.15.39:0/1920621706 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c0757b0 msgr2=0x7fe50c19c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c0757b0 0x7fe50c19c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 -- 172.21.15.39:0/1920621706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c10ff10 msgr2=0x7fe50c1a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c10ff10 0x7fe50c1a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe503e00640 1 -- 172.21.15.39:0/1920621706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1a2490 con 0x7fe50c076100 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe50a800640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c10ff10 0x7fe50c1a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.740+0000 7fe509e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c0757b0 0x7fe50c19c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:23:02.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.741+0000 7fe503e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 0x7fe50c19cac0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400e9c0 tx=0x7fe4f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.741+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f400cd80 con 0x7fe50c076100 2024-09-21T01:23:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.741+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c1a2780 con 0x7fe50c076100 2024-09-21T01:23:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.741+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f40035f0 con 0x7fe50c076100 2024-09-21T01:23:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.741+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4010690 con 0x7fe50c076100 2024-09-21T01:23:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.741+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c1a8ee0 con 0x7fe50c076100 2024-09-21T01:23:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.742+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe50c075bc0 con 0x7fe50c076100 2024-09-21T01:23:02.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.744+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fe4f4004000 con 0x7fe50c076100 2024-09-21T01:23:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.746+0000 7fe502a00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe4f00b6700 0x7fe4f00b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.746+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9511+0+0 (secure 0 0 0) 0x7fe4f40e8140 con 0x7fe50c076100 2024-09-21T01:23:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.746+0000 7fe509e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe4f00b6700 0x7fe4f00b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.747+0000 7fe509e00640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe4f00b6700 0x7fe4f00b8c10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009fc0 tx=0x7fe4fc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.748+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe4f40ae250 con 0x7fe50c076100 2024-09-21T01:23:02.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89808 13.86 MB/sec execute 209 sec latency 10272.010 ms 2024-09-21T01:23:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.985+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe50c19d650 con 0x7fe50c076100 2024-09-21T01:23:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.986+0000 7fe502a00640 1 -- 172.21.15.39:0/1920621706 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11912 (secure 0 0 0) 0x7fe50c19d650 con 0x7fe50c076100 2024-09-21T01:23:02.988 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:23:02.988 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":56,"btime":"2024-09-21T01:21:44:738665+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:44.386909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:23:02.988 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 56 2024-09-21T01:23:02.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.988+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe4f00b6700 msgr2=0x7fe4f00b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:02.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.988+0000 7fe50b200640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe4f00b6700 0x7fe4f00b8c10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009fc0 tx=0x7fe4fc005f90 comp rx=0 tx=0).stop 2024-09-21T01:23:02.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.988+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 msgr2=0x7fe50c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:02.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.988+0000 7fe50b200640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 0x7fe50c19cac0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400e9c0 tx=0x7fe4f400ee80 comp rx=0 tx=0).stop 2024-09-21T01:23:02.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 shutdown_connections 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe4f00b6700 0x7fe4f00b8c10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe50c10ff10 0x7fe50c1a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe50c076100 0x7fe50c19cac0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 --2- 172.21.15.39:0/1920621706 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe50c0757b0 0x7fe50c19c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 >> 172.21.15.39:0/1920621706 conn(0x7fe50c0ff440 msgr2=0x7fe50c109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 shutdown_connections 2024-09-21T01:23:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:02.989+0000 7fe50b200640 1 -- 172.21.15.39:0/1920621706 wait complete. 2024-09-21T01:23:03.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89808 13.79 MB/sec execute 210 sec latency 11272.145 ms 2024-09-21T01:23:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:03 smithi039 ceph-mon[30415]: pgmap v1001: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 414 KiB/s wr, 101 op/s 2024-09-21T01:23:03.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:03 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1920621706' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:04.030 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id edd8abad --release --await' 2024-09-21T01:23:04.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id edd8abad --release --await 2024-09-21T01:23:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:03 smithi150 ceph-mon[34945]: pgmap v1001: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 414 KiB/s wr, 101 op/s 2024-09-21T01:23:04.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:03 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1920621706' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:03 smithi160 ceph-mon[34696]: pgmap v1001: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 414 KiB/s wr, 101 op/s 2024-09-21T01:23:04.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:03 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1920621706' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:04.407 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:23:04.816 INFO:tasks.workunit.client.0.smithi039.stdout: 10 89808 13.73 MB/sec execute 211 sec latency 12272.513 ms 2024-09-21T01:23:05.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.078+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1566489447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 msgr2=0x7fa5cc10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.078+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1566489447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 0x7fa5cc10b900 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009900 tx=0x7fa5b802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1566489447 shutdown_connections 2024-09-21T01:23:05.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1566489447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc10be40 0x7fa5cc112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1566489447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 0x7fa5cc10b900 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1566489447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1566489447 >> 172.21.15.39:0/1566489447 conn(0x7fa5cc101680 msgr2=0x7fa5cc103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:05.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1566489447 shutdown_connections 2024-09-21T01:23:05.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.079+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1566489447 wait complete. 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 Processor -- start 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 -- start start 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 0x7fa5cc19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5cc10be40 0x7fa5cc1a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc115b00 con 0x7fa5cc105bb0 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc115980 con 0x7fa5cc10b4f0 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.080+0000 7fa5d2a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc115c80 con 0x7fa5cc10be40 2024-09-21T01:23:05.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc19c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48820/0 (socket says 172.21.15.39:48820) 2024-09-21T01:23:05.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 -- 172.21.15.39:0/1582867067 learned_addr learned my addr 172.21.15.39:0/1582867067 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:05.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d2000640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5cc10be40 0x7fa5cc1a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 0x7fa5cc19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 -- 172.21.15.39:0/1582867067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5cc10be40 msgr2=0x7fa5cc1a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5cc10be40 0x7fa5cc1a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 -- 172.21.15.39:0/1582867067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 msgr2=0x7fa5cc19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 0x7fa5cc19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.081+0000 7fa5d1600640 1 -- 172.21.15.39:0/1582867067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc1a2460 con 0x7fa5cc105bb0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.082+0000 7fa5d1600640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc19c4f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e9f0 tx=0x7fa5c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.082+0000 7fa5ca000640 1 -- 172.21.15.39:0/1582867067 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000cd80 con 0x7fa5cc105bb0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.082+0000 7fa5ca000640 1 -- 172.21.15.39:0/1582867067 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c00054d0 con 0x7fa5cc105bb0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.082+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc1a2750 con 0x7fa5cc105bb0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.082+0000 7fa5ca000640 1 -- 172.21.15.39:0/1582867067 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0010690 con 0x7fa5cc105bb0 2024-09-21T01:23:05.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.082+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc1a8eb0 con 0x7fa5cc105bb0 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 Processor -- start 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 -- start start 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5ca000640 1 -- 172.21.15.39:0/1582867067 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa5c0004e90 con 0x7fa5cc105bb0 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01e9a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01aca660 0x559b01e9a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01ad2840 0x559b01e9adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559b01e11290 con 0x559b01ad2840 2024-09-21T01:23:05.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559b01e006c0 con 0x559b01af3520 2024-09-21T01:23:05.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5dadde740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b01e044d0 con 0x559b01aca660 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.086+0000 7fa5d0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01ad2840 0x559b01e9adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d1600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01aca660 0x559b01e9a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01e9a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01e9a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43792/0 (socket says 172.21.15.39:43792) 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 -- 172.21.15.39:0/981677233 learned_addr learned my addr 172.21.15.39:0/981677233 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 -- 172.21.15.39:0/981677233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01aca660 msgr2=0x559b01e9a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01aca660 0x559b01e9a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 -- 172.21.15.39:0/981677233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01ad2840 msgr2=0x559b01e9adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01ad2840 0x559b01e9adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 -- 172.21.15.39:0/981677233 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b01d8d460 con 0x559b01af3520 2024-09-21T01:23:05.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.087+0000 7fa5d2000640 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01e9a340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009fd0 tx=0x7fa5c40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d45839b889f678 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:05.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.088+0000 7fa5be600640 1 -- 172.21.15.39:0/981677233 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004d20 con 0x559b01af3520 2024-09-21T01:23:05.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.088+0000 7fa5be600640 1 -- 172.21.15.39:0/981677233 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c4004050 con 0x559b01af3520 2024-09-21T01:23:05.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.088+0000 7fa5be600640 1 -- 172.21.15.39:0/981677233 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4005220 con 0x559b01af3520 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.088+0000 7fa5ca000640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa5b40b6680 0x7fa5b40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.088+0000 7fa5ca000640 1 -- 172.21.15.39:0/1582867067 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7fa5c00e7ff0 con 0x7fa5cc105bb0 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.088+0000 7fa5d1600640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa5b40b6680 0x7fa5b40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 -- 172.21.15.39:0/981677233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 msgr2=0x559b01e9a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01e9a340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009fd0 tx=0x7fa5c40118d0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 -- 172.21.15.39:0/981677233 shutdown_connections 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01ad2840 0x559b01e9adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01aca660 0x559b01e9a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 --2- 172.21.15.39:0/981677233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01e9a340 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 -- 172.21.15.39:0/981677233 >> 172.21.15.39:0/981677233 conn(0x559b01ae7910 msgr2=0x559b01e98230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 -- 172.21.15.39:0/981677233 shutdown_connections 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5dadde740 1 -- 172.21.15.39:0/981677233 wait complete. 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.089+0000 7fa5d1600640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa5b40b6680 0x7fa5b40b8b90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x559b01e98fc0 tx=0x7fa5c00f6b80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 -- start start 2024-09-21T01:23:05.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 Processor -- start 2024-09-21T01:23:05.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01ea5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01ea63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01eb1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559b01c61bb0 con 0x559b01aca660 2024-09-21T01:23:05.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559b01d9b6c0 con 0x559b01af3520 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b01ea8b20 con 0x559b01ad2840 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.090+0000 7fa5dadde740 10 client.0 _ll_register_callbacks cb 0x559b01db6490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01ea5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01ea63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01ea63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43058/0 (socket says 172.21.15.39:43058) 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d0c00640 1 -- 172.21.15.39:0/2955883431 learned_addr learned my addr 172.21.15.39:0/2955883431 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01eb1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 -- 172.21.15.39:0/2955883431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 msgr2=0x559b01ea63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01ea63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 -- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 msgr2=0x559b01eb1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01eb1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d2000640 1 -- 172.21.15.39:0/2955883431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b01b5eb00 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.091+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01ea63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5d2000640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01ea5e80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa5c401b040 tx=0x7fa5c4003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24dec76ca89b16d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5b2a00640 1 -- 172.21.15.39:0/2955883431 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004e50 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5c4004050 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5b2a00640 1 -- 172.21.15.39:0/2955883431 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7fa5c4005040 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5b2a00640 1 -- 172.21.15.39:0/2955883431 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fa5c4035050 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5dadde740 10 client.16323 fetch_fsmap learned FSMap version 56 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.092+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x559b01bd5960 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5b2a00640 1 -- 172.21.15.39:0/2955883431 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c4004050 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5b2a00640 1 -- 172.21.15.39:0/2955883431 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fa5c400f670 con 0x559b01aca660 2024-09-21T01:23:05.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5dadde740 10 client.16323 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:23:05.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5dadde740 10 client.16323 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:23:05.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5dadde740 20 client.16323 populate_metadata read hostname 'smithi039' 2024-09-21T01:23:05.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01ab1ea0 0x559b01eb7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5dadde740 4 client.16323 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:23:05.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b01e131e0 con 0x559b01ab1ea0 2024-09-21T01:23:05.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.093+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01ab1ea0 0x559b01eb7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.094+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01ab1ea0 0x559b01eb7cd0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3709a7c059a2bae6 server_cookie=fbb26e200ff20990 in_seq=0 out_seq=0 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.094+0000 7fa5b2a00640 10 client.16323 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5b2a00640 1 -- 172.21.15.39:0/2955883431 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559b01e131e0 con 0x559b01ab1ea0 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5b2a00640 10 client.16323 handle_command_reply: tid=0 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 client.16323 shutdown 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 msgr2=0x559b01ea5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01ea5e80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa5c401b040 tx=0x7fa5c4003f50 comp rx=0 tx=0).stop 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 shutdown_connections 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01ab1ea0 0x559b01eb7cd0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01eb1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01ea63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2955883431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01ea5e80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.095+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 >> 172.21.15.39:0/2955883431 conn(0x559b01ae7910 msgr2=0x559b01e9d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.096+0000 7fa5b2a00640 0 client.16323 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.096+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 shutdown_connections 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.096+0000 7fa5dadde740 1 -- 172.21.15.39:0/2955883431 wait complete. 2024-09-21T01:23:05.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.096+0000 7fa5dadde740 20 client.16323 trim_cache size 0 max 16384 2024-09-21T01:23:05.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.114+0000 7fa5dadde740 1 Processor -- start 2024-09-21T01:23:05.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.114+0000 7fa5dadde740 1 -- start start 2024-09-21T01:23:05.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01aca660 0x559b01eb1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eb2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01af3520 0x559b01e9f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5dadde740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1b1650 con 0x559b01af3520 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5dadde740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1aeef0 con 0x559b01aca660 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5dadde740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1acd60 con 0x559b01ad2840 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01aca660 0x559b01eb1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eb2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01aca660 0x559b01eb1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43804/0 (socket says 172.21.15.39:43804) 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d2000640 1 -- 172.21.15.39:0/2115055156 learned_addr learned my addr 172.21.15.39:0/2115055156 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d2000640 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01af3520 0x559b01e9f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d0c00640 1 -- 172.21.15.39:0/2115055156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01aca660 msgr2=0x559b01eb1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01aca660 0x559b01eb1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d0c00640 1 -- 172.21.15.39:0/2115055156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01af3520 msgr2=0x559b01e9f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01af3520 0x559b01e9f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.115+0000 7fa5d0c00640 1 -- 172.21.15.39:0/2115055156 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b01e97960 con 0x559b01ad2840 2024-09-21T01:23:05.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.116+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eb2280 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8031440 tx=0x7fa5b8005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8eeb7bffd0b17f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:05.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.116+0000 7fa5bdc00640 1 -- 172.21.15.39:0/2115055156 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8049020 con 0x559b01ad2840 2024-09-21T01:23:05.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5bdc00640 1 -- 172.21.15.39:0/2115055156 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b8005040 con 0x559b01ad2840 2024-09-21T01:23:05.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5bdc00640 1 -- 172.21.15.39:0/2115055156 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035040 con 0x559b01ad2840 2024-09-21T01:23:05.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 -- 172.21.15.39:0/2115055156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 msgr2=0x559b01eb2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eb2280 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8031440 tx=0x7fa5b8005a20 comp rx=0 tx=0).stop 2024-09-21T01:23:05.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 -- 172.21.15.39:0/2115055156 shutdown_connections 2024-09-21T01:23:05.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01af3520 0x559b01e9f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eb2280 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 --2- 172.21.15.39:0/2115055156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01aca660 0x559b01eb1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.117+0000 7fa5dadde740 1 -- 172.21.15.39:0/2115055156 >> 172.21.15.39:0/2115055156 conn(0x559b01ae7910 msgr2=0x559b01e9da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.118+0000 7fa5dadde740 1 -- 172.21.15.39:0/2115055156 shutdown_connections 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.118+0000 7fa5dadde740 1 -- 172.21.15.39:0/2115055156 wait complete. 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.118+0000 7fa5dadde740 1 -- start start 2024-09-21T01:23:05.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.118+0000 7fa5dadde740 1 Processor -- start 2024-09-21T01:23:05.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01eaddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01ea8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ad3b0 con 0x559b01aca660 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ae820 con 0x559b01af3520 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ada80 con 0x559b01ad2840 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5dadde740 10 client.0 _ll_register_callbacks cb 0x7fa5c40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01eaddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01eaddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48856/0 (socket says 172.21.15.39:48856) 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 -- 172.21.15.39:0/591774415 learned_addr learned my addr 172.21.15.39:0/591774415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01ea8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 -- 172.21.15.39:0/591774415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 msgr2=0x559b01eae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 -- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 msgr2=0x559b01ea8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01ea8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.119+0000 7fa5d2000640 1 -- 172.21.15.39:0/591774415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b01ea92a0 con 0x559b01aca660 2024-09-21T01:23:05.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5d2000640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01eaddb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4014040 tx=0x7fa5c4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9beff094db7a3591 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5b2a00640 1 -- 172.21.15.39:0/591774415 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004e60 con 0x559b01aca660 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b8005040 con 0x559b01aca660 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5b2a00640 1 -- 172.21.15.39:0/591774415 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7fa5c400f4e0 con 0x559b01aca660 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5b2a00640 1 -- 172.21.15.39:0/591774415 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fa5c404a050 con 0x559b01aca660 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5dadde740 10 client.16329 fetch_fsmap learned FSMap version 56 2024-09-21T01:23:05.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.120+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x559b01f22ce0 con 0x559b01aca660 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5b2a00640 1 -- 172.21.15.39:0/591774415 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b8005040 con 0x559b01aca660 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5b2a00640 1 -- 172.21.15.39:0/591774415 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7fa5c4019630 con 0x559b01aca660 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5dadde740 10 client.16329 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5dadde740 10 client.16329 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5dadde740 20 client.16329 populate_metadata read hostname 'smithi039' 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5dadde740 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01a87250 0x559b01f252f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5dadde740 4 client.16329 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "edd8abad", "release": true, "await": true}] 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "edd8abad", "release": true, "await": true}) -- 0x559b01e0ec90 con 0x559b01a87250 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.121+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01a87250 0x559b01f252f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:05.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.122+0000 7fa5d0c00640 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01a87250 0x559b01f252f0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2691bd4139f9a2a6 server_cookie=c73e0f5cd9d02319 in_seq=0 out_seq=0 2024-09-21T01:23:05.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.122+0000 7fa5b2a00640 10 client.16329 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:05.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.160+0000 7fa5b2a00640 1 -- 172.21.15.39:0/591774415 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559b01e0ec90 con 0x559b01a87250 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.160+0000 7fa5b2a00640 10 client.16329 handle_command_reply: tid=0 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.160+0000 7fa5dadde740 1 client.16329 shutdown 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 msgr2=0x559b01eaddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01eaddb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4014040 tx=0x7fa5c4009da0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 shutdown_connections 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b01a87250 0x559b01f252f0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b01af3520 0x559b01ea8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b01ad2840 0x559b01eae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5b2a00640 0 client.16329 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 --2- 172.21.15.39:0/591774415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b01aca660 0x559b01eaddb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.161+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 >> 172.21.15.39:0/591774415 conn(0x559b01ae7910 msgr2=0x559b01e9da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:05.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.162+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 shutdown_connections 2024-09-21T01:23:05.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.162+0000 7fa5dadde740 1 -- 172.21.15.39:0/591774415 wait complete. 2024-09-21T01:23:05.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.162+0000 7fa5dadde740 20 client.16329 trim_cache size 0 max 16384 2024-09-21T01:23:05.164 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:23:05.164 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 56, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 76, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "edd8abad": { 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 76, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 142.2, 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:23:05.165 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:05.166 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa5b40b6680 msgr2=0x7fa5b40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa5b40b6680 0x7fa5b40b8b90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x559b01e98fc0 tx=0x7fa5c00f6b80 comp rx=0 tx=0).stop 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 msgr2=0x7fa5cc19c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc19c4f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e9f0 tx=0x7fa5c000edf0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 shutdown_connections 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5cc10be40 0x7fa5cc1a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa5b40b6680 0x7fa5b40b8b90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5cc10b4f0 0x7fa5cc19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 --2- 172.21.15.39:0/1582867067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5cc105bb0 0x7fa5cc19c4f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:05.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.163+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 >> 172.21.15.39:0/1582867067 conn(0x7fa5cc101680 msgr2=0x7fa5cc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.164+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 shutdown_connections 2024-09-21T01:23:05.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:05.164+0000 7fa5d2a00640 1 -- 172.21.15.39:0/1582867067 wait complete. 2024-09-21T01:23:05.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 90263 13.82 MB/sec execute 212 sec latency 12617.239 ms 2024-09-21T01:23:06.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:05 smithi150 ceph-mon[34945]: pgmap v1002: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 293 KiB/s wr, 76 op/s 2024-09-21T01:23:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:05 smithi150 ceph-mon[34945]: osdmap e117: 12 total, 12 up, 12 in 2024-09-21T01:23:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:05 smithi160 ceph-mon[34696]: pgmap v1002: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 293 KiB/s wr, 76 op/s 2024-09-21T01:23:06.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:05 smithi160 ceph-mon[34696]: osdmap e117: 12 total, 12 up, 12 in 2024-09-21T01:23:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:05 smithi039 ceph-mon[30415]: pgmap v1002: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 293 KiB/s wr, 76 op/s 2024-09-21T01:23:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:05 smithi039 ceph-mon[30415]: osdmap e117: 12 total, 12 up, 12 in 2024-09-21T01:23:06.271 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'edd8abad', total seconds elapsed: 31.7 2024-09-21T01:23:06.271 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.15 2024-09-21T01:23:06.271 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:23:06.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 90920 13.93 MB/sec execute 213 sec latency 128.883 ms 2024-09-21T01:23:06.889 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:23:07.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 91382 13.98 MB/sec execute 214 sec latency 384.103 ms 2024-09-21T01:23:07.882 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:07 smithi039 ceph-mon[30415]: pgmap v1004: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 153 KiB/s wr, 39 op/s 2024-09-21T01:23:08.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:07 smithi150 ceph-mon[34945]: pgmap v1004: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 153 KiB/s wr, 39 op/s 2024-09-21T01:23:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:07 smithi160 ceph-mon[34696]: pgmap v1004: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 153 KiB/s wr, 39 op/s 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 -- 172.21.15.39:0/4060703699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67441094d0 msgr2=0x7f67441098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 --2- 172.21.15.39:0/4060703699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67441094d0 0x7f67441098e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6730009900 tx=0x7f673002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 -- 172.21.15.39:0/4060703699 shutdown_connections 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 --2- 172.21.15.39:0/4060703699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744109e20 0x7f6744110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 --2- 172.21.15.39:0/4060703699 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f67441094d0 0x7f67441098e0 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6730009900 tx=0x7f673002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 --2- 172.21.15.39:0/4060703699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6744103b90 0x7f6744103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.763+0000 7f6749400640 1 -- 172.21.15.39:0/4060703699 >> 172.21.15.39:0/4060703699 conn(0x7f67440ff5e0 msgr2=0x7f6744101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:08.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.764+0000 7f6749400640 1 -- 172.21.15.39:0/4060703699 shutdown_connections 2024-09-21T01:23:08.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.765+0000 7f6749400640 1 -- 172.21.15.39:0/4060703699 wait complete. 2024-09-21T01:23:08.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.767+0000 7f6749400640 1 Processor -- start 2024-09-21T01:23:08.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.767+0000 7f6749400640 1 -- start start 2024-09-21T01:23:08.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.768+0000 7f6749400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 0x7f6744074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:08.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.768+0000 7f6749400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6744109e20 0x7f674406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:08.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.768+0000 7f6749400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f674406e7e0 0x7f6744073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:08.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.768+0000 7f6742800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 0x7f6744074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:08.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6742800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 0x7f6744074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43828/0 (socket says 172.21.15.39:43828) 2024-09-21T01:23:08.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6749400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6744113b30 con 0x7f6744109e20 2024-09-21T01:23:08.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6749400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67441139b0 con 0x7f6744103b90 2024-09-21T01:23:08.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6749400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6744113cb0 con 0x7f674406e7e0 2024-09-21T01:23:08.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6742800640 1 -- 172.21.15.39:0/1715918761 learned_addr learned my addr 172.21.15.39:0/1715918761 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6743200640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f674406e7e0 0x7f6744073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.769+0000 7f6741e00640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6744109e20 0x7f674406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.770+0000 7f6742800640 1 -- 172.21.15.39:0/1715918761 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f674406e7e0 msgr2=0x7f6744073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.770+0000 7f6742800640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f674406e7e0 0x7f6744073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.770+0000 7f6742800640 1 -- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6744109e20 msgr2=0x7f674406e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.770+0000 7f6742800640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6744109e20 0x7f674406e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:08.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.770+0000 7f6742800640 1 -- 172.21.15.39:0/1715918761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674406efa0 con 0x7f6744103b90 2024-09-21T01:23:08.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.771+0000 7f6742800640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 0x7f6744074fc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f673400b2f0 tx=0x7f673400b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:08.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.772+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734022070 con 0x7f6744103b90 2024-09-21T01:23:08.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.772+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673401b9d0 con 0x7f6744103b90 2024-09-21T01:23:08.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.772+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673401da90 con 0x7f6744103b90 2024-09-21T01:23:08.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.772+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6744074170 con 0x7f6744103b90 2024-09-21T01:23:08.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.773+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6744074630 con 0x7f6744103b90 2024-09-21T01:23:08.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.775+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6744104e10 con 0x7f6744103b90 2024-09-21T01:23:08.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.778+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6734025070 con 0x7f6744103b90 2024-09-21T01:23:08.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.780+0000 7f673b400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f67280b6850 0x7f67280b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:08.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.781+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7f67340ef7a0 con 0x7f6744103b90 2024-09-21T01:23:08.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.782+0000 7f6741e00640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f67280b6850 0x7f67280b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:08.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.783+0000 7f6741e00640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f67280b6850 0x7f67280b8d60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6730008000 tx=0x7f6730000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:08.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:08.786+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f67340b9c40 con 0x7f6744103b90 2024-09-21T01:23:08.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 91953 14.04 MB/sec execute 215 sec latency 103.348 ms 2024-09-21T01:23:09.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.219+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6744074950 con 0x7f6744103b90 2024-09-21T01:23:09.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.220+0000 7f673b400640 1 -- 172.21.15.39:0/1715918761 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11912 (secure 0 0 0) 0x7f67340beb80 con 0x7f6744103b90 2024-09-21T01:23:09.221 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:23:09.221 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":56,"btime":"2024-09-21T01:21:44:738665+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:44.386909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:23:09.221 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 56 2024-09-21T01:23:09.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.224+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f67280b6850 msgr2=0x7f67280b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:09.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.224+0000 7f6749400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f67280b6850 0x7f67280b8d60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6730008000 tx=0x7f6730000bc0 comp rx=0 tx=0).stop 2024-09-21T01:23:09.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.224+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 msgr2=0x7f6744074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:09.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.224+0000 7f6749400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 0x7f6744074fc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f673400b2f0 tx=0x7f673400b7b0 comp rx=0 tx=0).stop 2024-09-21T01:23:09.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 shutdown_connections 2024-09-21T01:23:09.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f674406e7e0 0x7f6744073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:09.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6744109e20 0x7f674406e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:09.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f67280b6850 0x7f67280b8d60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:09.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 --2- 172.21.15.39:0/1715918761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6744103b90 0x7f6744074fc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:09.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 >> 172.21.15.39:0/1715918761 conn(0x7f67440ff5e0 msgr2=0x7f6744101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:09.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.225+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 shutdown_connections 2024-09-21T01:23:09.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:09.227+0000 7f6749400640 1 -- 172.21.15.39:0/1715918761 wait complete. 2024-09-21T01:23:09.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 92476 14.10 MB/sec execute 216 sec latency 89.119 ms 2024-09-21T01:23:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:09 smithi150 ceph-mon[34945]: pgmap v1005: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 99 op/s 2024-09-21T01:23:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:09 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1715918761' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:09 smithi160 ceph-mon[34696]: pgmap v1005: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 99 op/s 2024-09-21T01:23:10.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:09 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1715918761' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:10.129 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:09 smithi039 ceph-mon[30415]: pgmap v1005: 129 pgs: 129 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 99 op/s 2024-09-21T01:23:10.129 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:09 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1715918761' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:10.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 93003 14.17 MB/sec execute 217 sec latency 251.005 ms 2024-09-21T01:23:11.160 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0' 2024-09-21T01:23:11.160 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 142.15 --await-for 45.0 2024-09-21T01:23:11.759 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:23:11.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 93631 14.28 MB/sec execute 218 sec latency 149.021 ms 2024-09-21T01:23:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:11 smithi150 ceph-mon[34945]: pgmap v1006: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-09-21T01:23:12.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:11 smithi039 ceph-mon[30415]: pgmap v1006: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-09-21T01:23:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:11 smithi160 ceph-mon[34696]: pgmap v1006: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-09-21T01:23:12.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94325 14.35 MB/sec execute 219 sec latency 138.335 ms 2024-09-21T01:23:12.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.842+0000 7f4622400640 1 -- 172.21.15.39:0/2663120323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109330 msgr2=0x7f461c109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:12.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.842+0000 7f4622400640 1 --2- 172.21.15.39:0/2663120323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109330 0x7f461c109740 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4608009900 tx=0x7f460802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.843+0000 7f4622400640 1 -- 172.21.15.39:0/2663120323 shutdown_connections 2024-09-21T01:23:12.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.843+0000 7f4622400640 1 --2- 172.21.15.39:0/2663120323 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c109c80 0x7f461c1104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.843+0000 7f4622400640 1 --2- 172.21.15.39:0/2663120323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109330 0x7f461c109740 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.843+0000 7f4622400640 1 --2- 172.21.15.39:0/2663120323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c1039f0 0x7f461c103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.843+0000 7f4622400640 1 -- 172.21.15.39:0/2663120323 >> 172.21.15.39:0/2663120323 conn(0x7f461c0ff480 msgr2=0x7f461c1018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:12.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.846+0000 7f4622400640 1 -- 172.21.15.39:0/2663120323 shutdown_connections 2024-09-21T01:23:12.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.846+0000 7f4622400640 1 -- 172.21.15.39:0/2663120323 wait complete. 2024-09-21T01:23:12.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.849+0000 7f4622400640 1 Processor -- start 2024-09-21T01:23:12.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.849+0000 7f4622400640 1 -- start start 2024-09-21T01:23:12.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.849+0000 7f4622400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c1039f0 0x7f461c1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.849+0000 7f4622400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c109330 0x7f461c1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.849+0000 7f4622400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109c80 0x7f461c10f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4621000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c1039f0 0x7f461c1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4621000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c1039f0 0x7f461c1a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:43112/0 (socket says 172.21.15.39:43112) 2024-09-21T01:23:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4621000640 1 -- 172.21.15.39:0/3039373769 learned_addr learned my addr 172.21.15.39:0/3039373769 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1139b0 con 0x7f461c109330 2024-09-21T01:23:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f461c113830 con 0x7f461c109c80 2024-09-21T01:23:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f461c113b30 con 0x7f461c1039f0 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f4621a00640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109c80 0x7f461c10f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.850+0000 7f461be00640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c109330 0x7f461c1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.851+0000 7f4621a00640 1 -- 172.21.15.39:0/3039373769 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c1039f0 msgr2=0x7f461c1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.851+0000 7f4621a00640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c1039f0 0x7f461c1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.851+0000 7f4621a00640 1 -- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c109330 msgr2=0x7f461c1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.851+0000 7f4621a00640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c109330 0x7f461c1a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.851+0000 7f4621a00640 1 -- 172.21.15.39:0/3039373769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c1101b0 con 0x7f461c109c80 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.851+0000 7f4621a00640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109c80 0x7f461c10f990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4614009fd0 tx=0x7f461400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.852+0000 7f4619600640 1 -- 172.21.15.39:0/3039373769 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461400ce20 con 0x7f461c109c80 2024-09-21T01:23:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.852+0000 7f4619600640 1 -- 172.21.15.39:0/3039373769 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4614005020 con 0x7f461c109c80 2024-09-21T01:23:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.852+0000 7f461be00640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c109330 0x7f461c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:23:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.852+0000 7f4619600640 1 -- 172.21.15.39:0/3039373769 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614010690 con 0x7f461c109c80 2024-09-21T01:23:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.852+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c1104a0 con 0x7f461c109c80 2024-09-21T01:23:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.852+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c19a840 con 0x7f461c109c80 2024-09-21T01:23:12.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.858+0000 7f4619600640 1 -- 172.21.15.39:0/3039373769 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4614010830 con 0x7f461c109c80 2024-09-21T01:23:12.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.860+0000 7f4619600640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46040b66f0 0x7f46040b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.861+0000 7f4619600640 1 -- 172.21.15.39:0/3039373769 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7f46140e92d0 con 0x7f461c109c80 2024-09-21T01:23:12.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.861+0000 7f4621000640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46040b66f0 0x7f46040b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.863+0000 7f462a6af740 1 Processor -- start 2024-09-21T01:23:12.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.864+0000 7f4621000640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46040b66f0 0x7f46040b8c00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f461c19e8f0 tx=0x7f4610008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:12.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.865+0000 7f462a6af740 1 -- start start 2024-09-21T01:23:12.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.866+0000 7f462a6af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22acc8850 0x55f22b06fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.866+0000 7f462a6af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b0700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f4621000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22acc8850 0x55f22b06fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f4621000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22acc8850 0x55f22b06fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53150/0 (socket says 172.21.15.39:53150) 2024-09-21T01:23:12.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f461be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b0700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f462a6af740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b070610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f462a6af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f22afe5a90 con 0x55f22acc8850 2024-09-21T01:23:12.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f462a6af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55f22afd4ec0 con 0x55f22ac9f990 2024-09-21T01:23:12.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f462a6af740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f22afd8cd0 con 0x55f22aca7b70 2024-09-21T01:23:12.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.867+0000 7f461be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b0700d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:54452/0 (socket says 172.21.15.39:54452) 2024-09-21T01:23:12.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.868+0000 7f461be00640 1 -- 172.21.15.39:0/535442421 learned_addr learned my addr 172.21.15.39:0/535442421 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:12.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.868+0000 7f4621a00640 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.869+0000 7f461be00640 1 -- 172.21.15.39:0/535442421 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 msgr2=0x55f22b070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.869+0000 7f461be00640 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b070610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.869+0000 7f461be00640 1 -- 172.21.15.39:0/535442421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22acc8850 msgr2=0x55f22b06fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.869+0000 7f461be00640 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22acc8850 0x55f22b06fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.869+0000 7f461be00640 1 -- 172.21.15.39:0/535442421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22af7a9d0 con 0x55f22ac9f990 2024-09-21T01:23:12.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.870+0000 7f461be00640 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b0700d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4608009900 tx=0x7f4608031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48ce75feb241a34 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:12.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.871+0000 7f460e000640 1 -- 172.21.15.39:0/535442421 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608035030 con 0x55f22ac9f990 2024-09-21T01:23:12.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.871+0000 7f460e000640 1 -- 172.21.15.39:0/535442421 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4608004460 con 0x55f22ac9f990 2024-09-21T01:23:12.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.871+0000 7f460e000640 1 -- 172.21.15.39:0/535442421 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608032ad0 con 0x55f22ac9f990 2024-09-21T01:23:12.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.872+0000 7f462a6af740 1 -- 172.21.15.39:0/535442421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 msgr2=0x55f22b0700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:12.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.872+0000 7f462a6af740 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b0700d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4608009900 tx=0x7f4608031400 comp rx=0 tx=0).stop 2024-09-21T01:23:12.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.874+0000 7f462a6af740 1 -- 172.21.15.39:0/535442421 shutdown_connections 2024-09-21T01:23:12.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.874+0000 7f462a6af740 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b070610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.874+0000 7f462a6af740 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b0700d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.874+0000 7f462a6af740 1 --2- 172.21.15.39:0/535442421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22acc8850 0x55f22b06fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.874+0000 7f462a6af740 1 -- 172.21.15.39:0/535442421 >> 172.21.15.39:0/535442421 conn(0x55f22acbcc40 msgr2=0x55f22b06da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:12.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.876+0000 7f462a6af740 1 -- 172.21.15.39:0/535442421 shutdown_connections 2024-09-21T01:23:12.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.876+0000 7f462a6af740 1 -- 172.21.15.39:0/535442421 wait complete. 2024-09-21T01:23:12.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.877+0000 7f462a6af740 1 -- start start 2024-09-21T01:23:12.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.878+0000 7f462a6af740 1 Processor -- start 2024-09-21T01:23:12.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.878+0000 7f462a6af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22ac9f990 0x55f22b07b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.878+0000 7f462a6af740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b07bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.878+0000 7f462a6af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.879+0000 7f462a6af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f22ae367b0 con 0x55f22ac9f990 2024-09-21T01:23:12.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.879+0000 7f462a6af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55f22af79650 con 0x55f22acc8850 2024-09-21T01:23:12.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.879+0000 7f462a6af740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f22b07e360 con 0x55f22aca7b70 2024-09-21T01:23:12.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.879+0000 7f462a6af740 10 client.0 _ll_register_callbacks cb 0x55f22ace00a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:23:12.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.879+0000 7f461be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b07bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.880+0000 7f4621a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.880+0000 7f4621000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22ac9f990 0x55f22b07b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.880+0000 7f4621000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22ac9f990 0x55f22b07b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53156/0 (socket says 172.21.15.39:53156) 2024-09-21T01:23:12.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.880+0000 7f4621000640 1 -- 172.21.15.39:0/2750901439 learned_addr learned my addr 172.21.15.39:0/2750901439 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:12.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.881+0000 7f4621a00640 1 -- 172.21.15.39:0/2750901439 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 msgr2=0x55f22b07bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.881+0000 7f4621a00640 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b07bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.881+0000 7f4621a00640 1 -- 172.21.15.39:0/2750901439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22ac9f990 msgr2=0x55f22b07b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.881+0000 7f4621a00640 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22ac9f990 0x55f22b07b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.881+0000 7f4621a00640 1 -- 172.21.15.39:0/2750901439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22ad42ba0 con 0x55f22acc8850 2024-09-21T01:23:12.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.882+0000 7f4621a00640 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f46140e65f0 tx=0x7f46140ae8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fdcf5512768b4ee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:12.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.883+0000 7f4602a00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:23:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.883+0000 7f4602a00640 1 -- 172.21.15.39:0/2750901439 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46140b8710 con 0x55f22acc8850 2024-09-21T01:23:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.884+0000 7f4602a00640 1 -- 172.21.15.39:0/2750901439 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7f46140054d0 con 0x55f22acc8850 2024-09-21T01:23:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.884+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4608004460 con 0x55f22acc8850 2024-09-21T01:23:12.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.885+0000 7f4602a00640 1 -- 172.21.15.39:0/2750901439 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f46140d20e0 con 0x55f22acc8850 2024-09-21T01:23:12.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.885+0000 7f462a6af740 10 client.26366 fetch_fsmap learned FSMap version 56 2024-09-21T01:23:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.885+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55f22b08a2f0 con 0x55f22acc8850 2024-09-21T01:23:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.886+0000 7f4602a00640 1 -- 172.21.15.39:0/2750901439 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4608004460 con 0x55f22acc8850 2024-09-21T01:23:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.888+0000 7f4602a00640 1 -- 172.21.15.39:0/2750901439 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f46140adf50 con 0x55f22acc8850 2024-09-21T01:23:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f462a6af740 10 client.26366 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:23:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f462a6af740 10 client.26366 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:23:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f462a6af740 20 client.26366 populate_metadata read hostname 'smithi039' 2024-09-21T01:23:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f462a6af740 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac871d0 0x55f22b08d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f462a6af740 4 client.26366 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:23:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f22afe79e0 con 0x55f22ac871d0 2024-09-21T01:23:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.889+0000 7f4621000640 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac871d0 0x55f22b08d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.891+0000 7f4602a00640 10 client.26366 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:12.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.891+0000 7f4621000640 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac871d0 0x55f22b08d890 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f3423d3f3689b69 server_cookie=fb2f7eef69b31385 in_seq=0 out_seq=0 2024-09-21T01:23:12.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.894+0000 7f4602a00640 1 -- 172.21.15.39:0/2750901439 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f22afe79e0 con 0x55f22ac871d0 2024-09-21T01:23:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.894+0000 7f4602a00640 10 client.26366 handle_command_reply: tid=0 2024-09-21T01:23:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.895+0000 7f462a6af740 1 client.26366 shutdown 2024-09-21T01:23:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.895+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 msgr2=0x55f22b087680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.895+0000 7f462a6af740 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f46140e65f0 tx=0x7f46140ae8c0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 shutdown_connections 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac871d0 0x55f22b08d890 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087680 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22aca7b70 0x55f22b07bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 --2- 172.21.15.39:0/2750901439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22ac9f990 0x55f22b07b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 >> 172.21.15.39:0/2750901439 conn(0x55f22acbcc40 msgr2=0x55f22b072d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f4602a00640 0 client.26366 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.896+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 shutdown_connections 2024-09-21T01:23:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.897+0000 7f462a6af740 1 -- 172.21.15.39:0/2750901439 wait complete. 2024-09-21T01:23:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.897+0000 7f462a6af740 20 client.26366 trim_cache size 0 max 16384 2024-09-21T01:23:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.938+0000 7f462a6af740 1 Processor -- start 2024-09-21T01:23:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.938+0000 7f462a6af740 1 -- start start 2024-09-21T01:23:12.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f462a6af740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 0x55f22b071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f462a6af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b087580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f462a6af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f462a6af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1c2670 con 0x55f22aca7b70 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f462a6af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1bff10 con 0x55f22acc8850 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f462a6af740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1bdd80 con 0x55f22ac9f990 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.939+0000 7f4621000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 0x55f22b071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.940+0000 7f4621000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 0x55f22b071580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:45662/0 (socket says 172.21.15.39:45662) 2024-09-21T01:23:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.940+0000 7f4621000640 1 -- 172.21.15.39:0/2679385289 learned_addr learned my addr 172.21.15.39:0/2679385289 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.940+0000 7f461be00640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b087580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.941+0000 7f4621a00640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.941+0000 7f4621000640 1 -- 172.21.15.39:0/2679385289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 msgr2=0x55f22b087ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.942+0000 7f4621000640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.942+0000 7f4621000640 1 -- 172.21.15.39:0/2679385289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 msgr2=0x55f22b087580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.942+0000 7f4621000640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b087580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.942+0000 7f4621000640 1 -- 172.21.15.39:0/2679385289 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22b07a480 con 0x55f22ac9f990 2024-09-21T01:23:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.942+0000 7f461be00640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b087580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:23:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.942+0000 7f4621a00640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:23:12.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.943+0000 7f4621000640 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 0x55f22b071580 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f461000d530 tx=0x7f4610013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8142ab087b0791f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:12.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.946+0000 7f460d600640 1 -- 172.21.15.39:0/2679385289 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461002e070 con 0x55f22ac9f990 2024-09-21T01:23:12.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.946+0000 7f460d600640 1 -- 172.21.15.39:0/2679385289 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4610014cb0 con 0x55f22ac9f990 2024-09-21T01:23:12.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.946+0000 7f460d600640 1 -- 172.21.15.39:0/2679385289 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610022ba0 con 0x55f22ac9f990 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 -- 172.21.15.39:0/2679385289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 msgr2=0x55f22b071580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 0x55f22b071580 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f461000d530 tx=0x7f4610013000 comp rx=0 tx=0).stop 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 -- 172.21.15.39:0/2679385289 shutdown_connections 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22acc8850 0x55f22b087ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b087580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 --2- 172.21.15.39:0/2679385289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22ac9f990 0x55f22b071580 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.947+0000 7f462a6af740 1 -- 172.21.15.39:0/2679385289 >> 172.21.15.39:0/2679385289 conn(0x55f22acbcc40 msgr2=0x55f22b0766d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.948+0000 7f462a6af740 1 -- 172.21.15.39:0/2679385289 shutdown_connections 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.948+0000 7f462a6af740 1 -- 172.21.15.39:0/2679385289 wait complete. 2024-09-21T01:23:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.949+0000 7f462a6af740 1 -- start start 2024-09-21T01:23:12.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.950+0000 7f462a6af740 1 Processor -- start 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.952+0000 7f462a6af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b083e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.952+0000 7f462a6af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b07e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.952+0000 7f462a6af740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22acc8850 0x55f22b07eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.952+0000 7f462a6af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1be3d0 con 0x55f22aca7b70 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.952+0000 7f462a6af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1bf840 con 0x55f22ac9f990 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.952+0000 7f462a6af740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1beaa0 con 0x55f22acc8850 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.953+0000 7f4621000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b083e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.953+0000 7f4621000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b083e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:54484/0 (socket says 172.21.15.39:54484) 2024-09-21T01:23:12.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.953+0000 7f4621000640 1 -- 172.21.15.39:0/2988928040 learned_addr learned my addr 172.21.15.39:0/2988928040 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:12.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.953+0000 7f461be00640 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b07e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.953+0000 7f462a6af740 10 client.0 _ll_register_callbacks cb 0x7f46140aab90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:23:12.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.954+0000 7f4621000640 1 -- 172.21.15.39:0/2988928040 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22acc8850 msgr2=0x55f22b07eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:23:12.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.954+0000 7f4621000640 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22acc8850 0x55f22b07eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.954+0000 7f4621000640 1 -- 172.21.15.39:0/2988928040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 msgr2=0x55f22b07e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:12.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.954+0000 7f4621000640 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b07e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:12.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.954+0000 7f4621000640 1 -- 172.21.15.39:0/2988928040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22b07f330 con 0x55f22ac9f990 2024-09-21T01:23:12.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.954+0000 7f4621000640 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b083e90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4610008010 tx=0x7f4610022780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab60c7961053874 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:12.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.955+0000 7f4602a00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:23:12.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.955+0000 7f4602a00640 1 -- 172.21.15.39:0/2988928040 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610037060 con 0x55f22ac9f990 2024-09-21T01:23:12.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.955+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4610014cb0 con 0x55f22ac9f990 2024-09-21T01:23:12.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.955+0000 7f4602a00640 1 -- 172.21.15.39:0/2988928040 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9599+0+0 (secure 0 0 0) 0x7f461000f040 con 0x55f22ac9f990 2024-09-21T01:23:12.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.956+0000 7f4602a00640 1 -- 172.21.15.39:0/2988928040 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f4610037060 con 0x55f22ac9f990 2024-09-21T01:23:12.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.956+0000 7f462a6af740 10 client.26369 fetch_fsmap learned FSMap version 56 2024-09-21T01:23:12.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.956+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55f22b0f9180 con 0x55f22ac9f990 2024-09-21T01:23:12.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.957+0000 7f4602a00640 1 -- 172.21.15.39:0/2988928040 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4610014cb0 con 0x55f22ac9f990 2024-09-21T01:23:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.957+0000 7f4602a00640 1 -- 172.21.15.39:0/2988928040 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f461003a050 con 0x55f22ac9f990 2024-09-21T01:23:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.958+0000 7f462a6af740 10 client.26369 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:23:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.958+0000 7f462a6af740 10 client.26369 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:23:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.958+0000 7f462a6af740 20 client.26369 populate_metadata read hostname 'smithi039' 2024-09-21T01:23:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.958+0000 7f462a6af740 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac5c580 0x55f22b0fb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.958+0000 7f462a6af740 4 client.26369 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}] 2024-09-21T01:23:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.958+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.15, "await_for": 45.0}) -- 0x55f22afe3490 con 0x55f22ac5c580 2024-09-21T01:23:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.959+0000 7f461be00640 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac5c580 0x55f22b0fb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.960+0000 7f4602a00640 10 client.26369 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:12.960+0000 7f461be00640 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac5c580 0x55f22b0fb850 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3442e298d54a8422 server_cookie=337e49fe025f158a in_seq=0 out_seq=0 2024-09-21T01:23:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.590+0000 7f4602a00640 1 -- 172.21.15.39:0/2988928040 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f22afe3490 con 0x55f22ac5c580 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.590+0000 7f4602a00640 10 client.26369 handle_command_reply: tid=0 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.590+0000 7f462a6af740 1 client.26369 shutdown 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 msgr2=0x55f22b083e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b083e90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4610008010 tx=0x7f4610022780 comp rx=0 tx=0).stop 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 shutdown_connections 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f22ac5c580 0x55f22b0fb850 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f22acc8850 0x55f22b07eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f22aca7b70 0x55f22b07e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 --2- 172.21.15.39:0/2988928040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f22ac9f990 0x55f22b083e90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f4602a00640 0 client.26369 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 >> 172.21.15.39:0/2988928040 conn(0x55f22acbcc40 msgr2=0x55f22b0766d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:13.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 shutdown_connections 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 1 -- 172.21.15.39:0/2988928040 wait complete. 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.591+0000 7f462a6af740 20 client.26369 trim_cache size 0 max 16384 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 56, 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 78, 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "b89eca93": { 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 78, 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:13.593 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 142.2, 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:13.594 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46040b66f0 msgr2=0x7f46040b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46040b66f0 0x7f46040b8c00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f461c19e8f0 tx=0x7f4610008040 comp rx=0 tx=0).stop 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109c80 msgr2=0x7f461c10f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109c80 0x7f461c10f990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4614009fd0 tx=0x7f461400ef10 comp rx=0 tx=0).stop 2024-09-21T01:23:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 shutdown_connections 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46040b66f0 0x7f46040b8c00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f461c109c80 0x7f461c10f990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f461c109330 0x7f461c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 --2- 172.21.15.39:0/3039373769 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f461c1039f0 0x7f461c1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 >> 172.21.15.39:0/3039373769 conn(0x7f461c0ff480 msgr2=0x7f461c107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 shutdown_connections 2024-09-21T01:23:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:13.592+0000 7f4622400640 1 -- 172.21.15.39:0/3039373769 wait complete. 2024-09-21T01:23:13.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 14.31 MB/sec execute 220 sec latency 805.872 ms 2024-09-21T01:23:14.022 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:13 smithi039 ceph-mon[30415]: pgmap v1007: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.9 MiB/s wr, 237 op/s 2024-09-21T01:23:14.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:13 smithi150 ceph-mon[34945]: pgmap v1007: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.9 MiB/s wr, 237 op/s 2024-09-21T01:23:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:13 smithi160 ceph-mon[34696]: pgmap v1007: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.9 MiB/s wr, 237 op/s 2024-09-21T01:23:14.685 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b89eca93', quiesce took 8.4 seconds. Will release after: 13.749999999999998 2024-09-21T01:23:14.817 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 14.24 MB/sec execute 221 sec latency 1805.940 ms 2024-09-21T01:23:15.714 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:15 smithi039 ceph-mon[30415]: pgmap v1008: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.6 MiB/s wr, 322 op/s 2024-09-21T01:23:15.714 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:15 smithi039 ceph-mon[30415]: osdmap e118: 12 total, 12 up, 12 in 2024-09-21T01:23:15.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 14.18 MB/sec execute 222 sec latency 2805.994 ms 2024-09-21T01:23:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:15 smithi150 ceph-mon[34945]: pgmap v1008: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.6 MiB/s wr, 322 op/s 2024-09-21T01:23:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:15 smithi150 ceph-mon[34945]: osdmap e118: 12 total, 12 up, 12 in 2024-09-21T01:23:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:15 smithi160 ceph-mon[34696]: pgmap v1008: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.6 MiB/s wr, 322 op/s 2024-09-21T01:23:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:15 smithi160 ceph-mon[34696]: osdmap e118: 12 total, 12 up, 12 in 2024-09-21T01:23:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 14.11 MB/sec execute 223 sec latency 3806.051 ms 2024-09-21T01:23:17.670 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:17 smithi039 ceph-mon[30415]: pgmap v1010: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.1 MiB/s wr, 351 op/s 2024-09-21T01:23:17.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 14.05 MB/sec execute 224 sec latency 4806.104 ms 2024-09-21T01:23:18.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:17 smithi150 ceph-mon[34945]: pgmap v1010: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.1 MiB/s wr, 351 op/s 2024-09-21T01:23:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:17 smithi160 ceph-mon[34696]: pgmap v1010: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.1 MiB/s wr, 351 op/s 2024-09-21T01:23:18.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.99 MB/sec execute 225 sec latency 5806.163 ms 2024-09-21T01:23:19.653 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:19 smithi039 ceph-mon[30415]: pgmap v1011: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.4 MiB/s wr, 375 op/s 2024-09-21T01:23:19.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.93 MB/sec execute 226 sec latency 6806.208 ms 2024-09-21T01:23:19.909 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:19 smithi160 ceph-mon[34696]: pgmap v1011: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.4 MiB/s wr, 375 op/s 2024-09-21T01:23:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:19 smithi150 ceph-mon[34945]: pgmap v1011: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.4 MiB/s wr, 375 op/s 2024-09-21T01:23:20.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.87 MB/sec execute 227 sec latency 7806.266 ms 2024-09-21T01:23:21.753 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:21 smithi039 ceph-mon[30415]: pgmap v1012: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.5 MiB/s wr, 310 op/s 2024-09-21T01:23:21.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.81 MB/sec execute 228 sec latency 8806.346 ms 2024-09-21T01:23:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:21 smithi150 ceph-mon[34945]: pgmap v1012: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.5 MiB/s wr, 310 op/s 2024-09-21T01:23:22.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:21 smithi160 ceph-mon[34696]: pgmap v1012: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.5 MiB/s wr, 310 op/s 2024-09-21T01:23:22.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.74 MB/sec execute 229 sec latency 9806.396 ms 2024-09-21T01:23:23.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.69 MB/sec execute 230 sec latency 10806.544 ms 2024-09-21T01:23:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:23 smithi150 ceph-mon[34945]: pgmap v1013: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.4 MiB/s wr, 278 op/s 2024-09-21T01:23:24.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:23 smithi160 ceph-mon[34696]: pgmap v1013: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.4 MiB/s wr, 278 op/s 2024-09-21T01:23:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:23 smithi039 ceph-mon[30415]: pgmap v1013: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.4 MiB/s wr, 278 op/s 2024-09-21T01:23:24.818 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.63 MB/sec execute 231 sec latency 11806.638 ms 2024-09-21T01:23:25.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.57 MB/sec execute 232 sec latency 12806.764 ms 2024-09-21T01:23:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:25 smithi150 ceph-mon[34945]: pgmap v1014: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.3 MiB/s wr, 213 op/s 2024-09-21T01:23:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:25 smithi160 ceph-mon[34696]: pgmap v1014: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.3 MiB/s wr, 213 op/s 2024-09-21T01:23:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:25 smithi039 ceph-mon[30415]: pgmap v1014: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.3 MiB/s wr, 213 op/s 2024-09-21T01:23:26.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.51 MB/sec execute 233 sec latency 13806.922 ms 2024-09-21T01:23:27.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.45 MB/sec execute 234 sec latency 14807.055 ms 2024-09-21T01:23:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:27 smithi150 ceph-mon[34945]: pgmap v1015: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 181 op/s 2024-09-21T01:23:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:27 smithi160 ceph-mon[34696]: pgmap v1015: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 181 op/s 2024-09-21T01:23:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:27 smithi039 ceph-mon[30415]: pgmap v1015: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 181 op/s 2024-09-21T01:23:28.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:23:28.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.39 MB/sec execute 235 sec latency 15807.108 ms 2024-09-21T01:23:28.867 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:28.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:28 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.014 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:23:29.014 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.015 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.015 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.015 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.015 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.015 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:28 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:23:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:28 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:29.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 -- 172.21.15.39:0/1493175216 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c109c40 msgr2=0x7f3b2c110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 --2- 172.21.15.39:0/1493175216 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c109c40 0x7f3b2c110470 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009be0 tx=0x7f3b2402a6c0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 -- 172.21.15.39:0/1493175216 shutdown_connections 2024-09-21T01:23:29.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 --2- 172.21.15.39:0/1493175216 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c109c40 0x7f3b2c110470 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 --2- 172.21.15.39:0/1493175216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c1092f0 0x7f3b2c109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 --2- 172.21.15.39:0/1493175216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 0x7f3b2c103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 -- 172.21.15.39:0/1493175216 >> 172.21.15.39:0/1493175216 conn(0x7f3b2c0ff440 msgr2=0x7f3b2c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:29.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 -- 172.21.15.39:0/1493175216 shutdown_connections 2024-09-21T01:23:29.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.468+0000 7f3b32a00640 1 -- 172.21.15.39:0/1493175216 wait complete. 2024-09-21T01:23:29.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.469+0000 7f3b32a00640 1 Processor -- start 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 -- start start 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 0x7f3b2c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c1092f0 0x7f3b2c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c109c40 0x7f3b2c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c113900 con 0x7f3b2c109c40 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c113780 con 0x7f3b2c1039b0 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c113a80 con 0x7f3b2c1092f0 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b31600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 0x7f3b2c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b30c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c1092f0 0x7f3b2c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b31600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 0x7f3b2c075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:60680/0 (socket says 172.21.15.39:60680) 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b31600640 1 -- 172.21.15.39:0/813618074 learned_addr learned my addr 172.21.15.39:0/813618074 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:29.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.470+0000 7f3b32000640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c109c40 0x7f3b2c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b30c00640 1 -- 172.21.15.39:0/813618074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 msgr2=0x7f3b2c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b30c00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 0x7f3b2c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b30c00640 1 -- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c109c40 msgr2=0x7f3b2c076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b30c00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c109c40 0x7f3b2c076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b30c00640 1 -- 172.21.15.39:0/813618074 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c1a85b0 con 0x7f3b2c1092f0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b32000640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c109c40 0x7f3b2c076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b30c00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c1092f0 0x7f3b2c075c50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c06b700 tx=0x7f3b1800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.471+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800ccd0 con 0x7f3b2c1092f0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.472+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c1a88a0 con 0x7f3b2c1092f0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.472+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b180035f0 con 0x7f3b2c1092f0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.472+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18010690 con 0x7f3b2c1092f0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.472+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c1a8e30 con 0x7f3b2c1092f0 2024-09-21T01:23:29.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.473+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c103dc0 con 0x7f3b2c1092f0 2024-09-21T01:23:29.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.475+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3b18003a80 con 0x7f3b2c1092f0 2024-09-21T01:23:29.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.477+0000 7f3b2a000640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b140b6680 0x7f3b140b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.477+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9511+0+0 (secure 0 0 0) 0x7f3b180e7df0 con 0x7f3b2c1092f0 2024-09-21T01:23:29.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.477+0000 7f3b31600640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b140b6680 0x7f3b140b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.478+0000 7f3b31600640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b140b6680 0x7f3b140b8b90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c077160 tx=0x7f3b20005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:29.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.479+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3b18009510 con 0x7f3b2c1092f0 2024-09-21T01:23:29.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.724+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b2c02e640 con 0x7f3b2c1092f0 2024-09-21T01:23:29.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.725+0000 7f3b2a000640 1 -- 172.21.15.39:0/813618074 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11912 (secure 0 0 0) 0x7f3b180adf90 con 0x7f3b2c1092f0 2024-09-21T01:23:29.727 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:23:29.727 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":56,"btime":"2024-09-21T01:21:44:738665+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:21:44.386909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:23:29.727 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 56 2024-09-21T01:23:29.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b140b6680 msgr2=0x7f3b140b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b140b6680 0x7f3b140b8b90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c077160 tx=0x7f3b20005f90 comp rx=0 tx=0).stop 2024-09-21T01:23:29.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c1092f0 msgr2=0x7f3b2c075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c1092f0 0x7f3b2c075c50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c06b700 tx=0x7f3b1800ee50 comp rx=0 tx=0).stop 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 shutdown_connections 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b2c109c40 0x7f3b2c076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3b2c1092f0 0x7f3b2c075c50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3b140b6680 0x7f3b140b8b90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 --2- 172.21.15.39:0/813618074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c1039b0 0x7f3b2c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 >> 172.21.15.39:0/813618074 conn(0x7f3b2c0ff440 msgr2=0x7f3b2c101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 shutdown_connections 2024-09-21T01:23:29.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:29.728+0000 7f3b32a00640 1 -- 172.21.15.39:0/813618074 wait complete. 2024-09-21T01:23:29.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.34 MB/sec execute 236 sec latency 16807.163 ms 2024-09-21T01:23:29.929 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:29 smithi039 ceph-mon[30415]: pgmap v1016: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 170 op/s 2024-09-21T01:23:29.933 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:29 smithi150 ceph-mon[34945]: pgmap v1016: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 170 op/s 2024-09-21T01:23:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:29 smithi160 ceph-mon[34696]: pgmap v1016: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 170 op/s 2024-09-21T01:23:30.798 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id b89eca93 --release --await' 2024-09-21T01:23:30.798 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id b89eca93 --release --await 2024-09-21T01:23:30.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.28 MB/sec execute 237 sec latency 17807.228 ms 2024-09-21T01:23:31.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:30 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/813618074' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:31.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:30 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:30 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/813618074' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:30 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:30 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/813618074' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:23:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:30 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:23:31.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94422 13.23 MB/sec execute 238 sec latency 18807.309 ms 2024-09-21T01:23:32.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 -- 172.21.15.39:0/3983763618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888410ff50 msgr2=0x7f8884112370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 --2- 172.21.15.39:0/3983763618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888410ff50 0x7f8884112370 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f887c009ca0 tx=0x7f887c02a8b0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 -- 172.21.15.39:0/3983763618 shutdown_connections 2024-09-21T01:23:32.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 --2- 172.21.15.39:0/3983763618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888410ff50 0x7f8884112370 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 --2- 172.21.15.39:0/3983763618 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88840760a0 0x7f888410fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 --2- 172.21.15.39:0/3983763618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8884075750 0x7f8884075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 -- 172.21.15.39:0/3983763618 >> 172.21.15.39:0/3983763618 conn(0x7f88840ff3e0 msgr2=0x7f8884101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 -- 172.21.15.39:0/3983763618 shutdown_connections 2024-09-21T01:23:32.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.040+0000 7f8888c00640 1 -- 172.21.15.39:0/3983763618 wait complete. 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.041+0000 7f8888c00640 1 Processor -- start 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.041+0000 7f8888c00640 1 -- start start 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8888c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8884075750 0x7f888419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8888c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 0x7f888419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8888c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888410ff50 0x7f88841a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8888c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8884115ac0 con 0x7f88840760a0 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8888c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8884115940 con 0x7f888410ff50 2024-09-21T01:23:32.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8888c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8884115c40 con 0x7f8884075750 2024-09-21T01:23:32.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8882a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 0x7f888419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8882a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 0x7f888419ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35672/0 (socket says 172.21.15.39:35672) 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8882a00640 1 -- 172.21.15.39:0/129874238 learned_addr learned my addr 172.21.15.39:0/129874238 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8883400640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8884075750 0x7f888419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8883e00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888410ff50 0x7f88841a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.042+0000 7f8882a00640 1 -- 172.21.15.39:0/129874238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8884075750 msgr2=0x7f888419c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8882a00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8884075750 0x7f888419c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8882a00640 1 -- 172.21.15.39:0/129874238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888410ff50 msgr2=0x7f88841a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8882a00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888410ff50 0x7f88841a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8882a00640 1 -- 172.21.15.39:0/129874238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88841a23e0 con 0x7f88840760a0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8883400640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8884075750 0x7f888419c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8882a00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 0x7f888419ca10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f887000e970 tx=0x7f887000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f887be00640 1 -- 172.21.15.39:0/129874238 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887000cd80 con 0x7f88840760a0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f887be00640 1 -- 172.21.15.39:0/129874238 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88700035f0 con 0x7f88840760a0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88841a26d0 con 0x7f88840760a0 2024-09-21T01:23:32.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f887be00640 1 -- 172.21.15.39:0/129874238 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870010690 con 0x7f88840760a0 2024-09-21T01:23:32.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.043+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88841a8e30 con 0x7f88840760a0 2024-09-21T01:23:32.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 Processor -- start 2024-09-21T01:23:32.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 -- start start 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f46a660 0x55624f83a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f83adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55624f7b1290 con 0x55624f493520 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55624f7a06c0 con 0x55624f46a660 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.048+0000 7f8890e45740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55624f7a44d0 con 0x55624f472840 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f887be00640 1 -- 172.21.15.39:0/129874238 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8870003aa0 con 0x7f88840760a0 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35674/0 (socket says 172.21.15.39:35674) 2024-09-21T01:23:32.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 -- 172.21.15.39:0/3368305868 learned_addr learned my addr 172.21.15.39:0/3368305868 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:32.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883400640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f46a660 0x55624f83a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8882a00640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f83adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 -- 172.21.15.39:0/3368305868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 msgr2=0x55624f83adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f83adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 -- 172.21.15.39:0/3368305868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f46a660 msgr2=0x55624f83a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f46a660 0x55624f83a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883e00640 1 -- 172.21.15.39:0/3368305868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55624f72d460 con 0x55624f493520 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8883400640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f46a660 0x55624f83a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.049+0000 7f8882a00640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f83adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.050+0000 7f8883e00640 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83a340 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f887c009800 tx=0x7f887c009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1fe3c346c7b139 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.050+0000 7f886fe00640 1 -- 172.21.15.39:0/3368305868 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c0305a0 con 0x55624f493520 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.050+0000 7f886fe00640 1 -- 172.21.15.39:0/3368305868 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f887c036030 con 0x55624f493520 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.050+0000 7f886fe00640 1 -- 172.21.15.39:0/3368305868 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c004280 con 0x55624f493520 2024-09-21T01:23:32.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f887be00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f88680b6680 0x7f88680b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f887be00640 1 -- 172.21.15.39:0/129874238 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9511+0+0 (secure 0 0 0) 0x7f88700e7e00 con 0x7f88840760a0 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 -- 172.21.15.39:0/3368305868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 msgr2=0x55624f83a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83a340 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f887c009800 tx=0x7f887c009830 comp rx=0 tx=0).stop 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8883400640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f88680b6680 0x7f88680b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 -- 172.21.15.39:0/3368305868 shutdown_connections 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f83adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f46a660 0x55624f83a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 --2- 172.21.15.39:0/3368305868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83a340 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 -- 172.21.15.39:0/3368305868 >> 172.21.15.39:0/3368305868 conn(0x55624f487910 msgr2=0x55624f838230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.051+0000 7f8890e45740 1 -- 172.21.15.39:0/3368305868 shutdown_connections 2024-09-21T01:23:32.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.052+0000 7f8890e45740 1 -- 172.21.15.39:0/3368305868 wait complete. 2024-09-21T01:23:32.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.052+0000 7f8883400640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f88680b6680 0x7f88680b8b90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x55624f838fc0 tx=0x7f8874005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.052+0000 7f8890e45740 1 -- start start 2024-09-21T01:23:32.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.052+0000 7f8890e45740 1 Processor -- start 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f845f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f846490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f851e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55624f601bb0 con 0x55624f46a660 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55624f73b6c0 con 0x55624f493520 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55624f848ac0 con 0x55624f472840 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8890e45740 10 client.0 _ll_register_callbacks cb 0x55624f756490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8883400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f845f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8883400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f845f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35684/0 (socket says 172.21.15.39:35684) 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8883400640 1 -- 172.21.15.39:0/2952242621 learned_addr learned my addr 172.21.15.39:0/2952242621 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:32.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.053+0000 7f8883e00640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f851e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8882a00640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f846490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8883400640 1 -- 172.21.15.39:0/2952242621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 msgr2=0x55624f846490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8883400640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f846490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8883400640 1 -- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 msgr2=0x55624f851e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8883400640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f851e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8883400640 1 -- 172.21.15.39:0/2952242621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55624f4feb00 con 0x55624f46a660 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f8883400640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f845f50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f887401eae0 tx=0x7f8874004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7911f72533a589 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.054+0000 7f886cc00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f886cc00640 1 -- 172.21.15.39:0/2952242621 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874005020 con 0x55624f46a660 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f886cc00640 1 -- 172.21.15.39:0/2952242621 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9511+0+0 (secure 0 0 0) 0x7f8874018030 con 0x55624f46a660 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f887c036030 con 0x55624f46a660 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f886cc00640 1 -- 172.21.15.39:0/2952242621 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f88740051c0 con 0x55624f46a660 2024-09-21T01:23:32.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f8890e45740 10 client.16362 fetch_fsmap learned FSMap version 56 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55624f575960 con 0x55624f46a660 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.055+0000 7f886cc00640 1 -- 172.21.15.39:0/2952242621 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f887c036030 con 0x55624f46a660 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f886cc00640 1 -- 172.21.15.39:0/2952242621 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f887400f5c0 con 0x55624f46a660 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8890e45740 10 client.16362 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8890e45740 10 client.16362 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8890e45740 20 client.16362 populate_metadata read hostname 'smithi039' 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8890e45740 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f451ea0 0x55624f857f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8890e45740 4 client.16362 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:23:32.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55624f7b31e0 con 0x55624f451ea0 2024-09-21T01:23:32.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.056+0000 7f8882a00640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f451ea0 0x55624f857f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.057+0000 7f8882a00640 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f451ea0 0x55624f857f20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b730d2ccf013a69 server_cookie=ebf5a8b5ea5baac8 in_seq=0 out_seq=0 2024-09-21T01:23:32.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.057+0000 7f886cc00640 10 client.16362 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:32.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f886cc00640 1 -- 172.21.15.39:0/2952242621 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55624f7b31e0 con 0x55624f451ea0 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f886cc00640 10 client.16362 handle_command_reply: tid=0 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 client.16362 shutdown 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 msgr2=0x55624f845f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f845f50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f887401eae0 tx=0x7f8874004730 comp rx=0 tx=0).stop 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 shutdown_connections 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f451ea0 0x55624f857f20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f851e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f846490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 --2- 172.21.15.39:0/2952242621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f845f50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 >> 172.21.15.39:0/2952242621 conn(0x55624f487910 msgr2=0x55624f83d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f886cc00640 0 client.16362 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 shutdown_connections 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.058+0000 7f8890e45740 1 -- 172.21.15.39:0/2952242621 wait complete. 2024-09-21T01:23:32.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.059+0000 7f8890e45740 20 client.16362 trim_cache size 0 max 16384 2024-09-21T01:23:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 Processor -- start 2024-09-21T01:23:32.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 -- start start 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f46a660 0x55624f840390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 0x55624f847c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f88841b5850 con 0x55624f493520 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f88841b30f0 con 0x55624f472840 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.078+0000 7f8890e45740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f88841b0f60 con 0x55624f46a660 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 0x55624f847c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 0x55624f847c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:60738/0 (socket says 172.21.15.39:60738) 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 -- 172.21.15.39:0/792685946 learned_addr learned my addr 172.21.15.39:0/792685946 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:32.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8883400640 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f46a660 0x55624f840390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 -- 172.21.15.39:0/792685946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f46a660 msgr2=0x55624f840390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f46a660 0x55624f840390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 -- 172.21.15.39:0/792685946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 msgr2=0x55624f83f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 -- 172.21.15.39:0/792685946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55624f846ca0 con 0x55624f472840 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.079+0000 7f8882a00640 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 0x55624f847c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f88700a9de0 tx=0x7f8870000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b23c63486c459079 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.080+0000 7f886f400640 1 -- 172.21.15.39:0/792685946 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700adf10 con 0x55624f472840 2024-09-21T01:23:32.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.080+0000 7f886f400640 1 -- 172.21.15.39:0/792685946 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88700b22f0 con 0x55624f472840 2024-09-21T01:23:32.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.080+0000 7f886f400640 1 -- 172.21.15.39:0/792685946 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700aca70 con 0x55624f472840 2024-09-21T01:23:32.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 -- 172.21.15.39:0/792685946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 msgr2=0x55624f847c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 0x55624f847c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f88700a9de0 tx=0x7f8870000bc0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 -- 172.21.15.39:0/792685946 shutdown_connections 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f493520 0x55624f83f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f472840 0x55624f847c30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 --2- 172.21.15.39:0/792685946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f46a660 0x55624f840390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 -- 172.21.15.39:0/792685946 >> 172.21.15.39:0/792685946 conn(0x55624f487910 msgr2=0x55624f840f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 -- 172.21.15.39:0/792685946 shutdown_connections 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 -- 172.21.15.39:0/792685946 wait complete. 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 -- start start 2024-09-21T01:23:32.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.081+0000 7f8890e45740 1 Processor -- start 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f84a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f84ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f8bb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f88841b15b0 con 0x55624f46a660 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f88841b2a20 con 0x55624f493520 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f88841b1c80 con 0x55624f472840 2024-09-21T01:23:32.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.082+0000 7f8890e45740 10 client.0 _ll_register_callbacks cb 0x7f887401e420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8882a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f84ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f8bb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f8bb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:60748/0 (socket says 172.21.15.39:60748) 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 -- 172.21.15.39:0/1822737103 learned_addr learned my addr 172.21.15.39:0/1822737103 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883400640 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f84a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 -- 172.21.15.39:0/1822737103 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 msgr2=0x55624f84ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f84ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 -- 172.21.15.39:0/1822737103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 msgr2=0x55624f84a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f84a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.083+0000 7f8883e00640 1 -- 172.21.15.39:0/1822737103 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55624f8bbf30 con 0x55624f493520 2024-09-21T01:23:32.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f8883e00640 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f8bb8f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f887c009620 tx=0x7f887c009950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=805c5d2ede640a26 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f886cc00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f886cc00640 1 -- 172.21.15.39:0/1822737103 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c030800 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88700b22f0 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f886cc00640 1 -- 172.21.15.39:0/1822737103 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9511+0+0 (secure 0 0 0) 0x7f887c0309a0 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f886cc00640 1 -- 172.21.15.39:0/1822737103 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f887c030ba0 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f8890e45740 10 client.26393 fetch_fsmap learned FSMap version 56 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55624f8cbab0 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.084+0000 7f886cc00640 1 -- 172.21.15.39:0/1822737103 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88700b22f0 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f886cc00640 1 -- 172.21.15.39:0/1822737103 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 56) ==== 8358+0+0 (secure 0 0 0) 0x7f887c043070 con 0x55624f493520 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8890e45740 10 client.26393 fetch_fsmap finished waiting for FSMap version 56 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8890e45740 10 client.26393 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8890e45740 20 client.26393 populate_metadata read hostname 'smithi039' 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8890e45740 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f427250 0x55624f8cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8890e45740 4 client.26393 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b89eca93", "release": true, "await": true}] 2024-09-21T01:23:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b89eca93", "release": true, "await": true}) -- 0x55624f7aec90 con 0x55624f427250 2024-09-21T01:23:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.085+0000 7f8883400640 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f427250 0x55624f8cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.086+0000 7f886cc00640 10 client.26393 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.086+0000 7f8883400640 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f427250 0x55624f8cde00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5910e67664d2a3b6 server_cookie=f0ed574f984cb4bf in_seq=0 out_seq=0 2024-09-21T01:23:32.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:31 smithi039 ceph-mon[30415]: pgmap v1017: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.4 MiB/s wr, 98 op/s 2024-09-21T01:23:32.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:31 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.125+0000 7f886cc00640 1 -- 172.21.15.39:0/1822737103 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55624f7aec90 con 0x55624f427250 2024-09-21T01:23:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.125+0000 7f886cc00640 10 client.26393 handle_command_reply: tid=0 2024-09-21T01:23:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.125+0000 7f8890e45740 1 client.26393 shutdown 2024-09-21T01:23:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 msgr2=0x55624f8bb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f8bb8f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f887c009620 tx=0x7f887c009950 comp rx=0 tx=0).stop 2024-09-21T01:23:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 shutdown_connections 2024-09-21T01:23:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55624f427250 0x55624f8cde00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55624f493520 0x55624f8bb8f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55624f472840 0x55624f84ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 --2- 172.21.15.39:0/1822737103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55624f46a660 0x55624f84a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 >> 172.21.15.39:0/1822737103 conn(0x55624f487910 msgr2=0x55624f840f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f886cc00640 0 client.26393 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 shutdown_connections 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 1 -- 172.21.15.39:0/1822737103 wait complete. 2024-09-21T01:23:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.126+0000 7f8890e45740 20 client.26393 trim_cache size 0 max 16384 2024-09-21T01:23:32.129 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 56, 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 80, 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "b89eca93": { 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 80, 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:23:32.130 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 142.2, 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:23:32.131 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.129+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f88680b6680 msgr2=0x7f88680b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.129+0000 7f8888c00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f88680b6680 0x7f88680b8b90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x55624f838fc0 tx=0x7f8874005f90 comp rx=0 tx=0).stop 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.129+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 msgr2=0x7f888419ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.129+0000 7f8888c00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 0x7f888419ca10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f887000e970 tx=0x7f887000ee30 comp rx=0 tx=0).stop 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.130+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 shutdown_connections 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.130+0000 7f8888c00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f88680b6680 0x7f88680b8b90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.130+0000 7f8888c00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888410ff50 0x7f88841a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.130+0000 7f8888c00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88840760a0 0x7f888419ca10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.130+0000 7f8888c00640 1 --2- 172.21.15.39:0/129874238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8884075750 0x7f888419c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.130+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 >> 172.21.15.39:0/129874238 conn(0x7f88840ff3e0 msgr2=0x7f8884109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.132+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 shutdown_connections 2024-09-21T01:23:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:23:32.132+0000 7f8888c00640 1 -- 172.21.15.39:0/129874238 wait complete. 2024-09-21T01:23:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:31 smithi150 ceph-mon[34945]: pgmap v1017: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.4 MiB/s wr, 98 op/s 2024-09-21T01:23:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:31 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:31 smithi160 ceph-mon[34696]: pgmap v1017: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.4 MiB/s wr, 98 op/s 2024-09-21T01:23:32.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:31 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 94822 13.28 MB/sec execute 239 sec latency 19121.482 ms 2024-09-21T01:23:33.161 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b89eca93', total seconds elapsed: 26.9 2024-09-21T01:23:33.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 95528 13.39 MB/sec execute 240 sec latency 70.703 ms 2024-09-21T01:23:34.156 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:33 smithi039 ceph-mon[30415]: pgmap v1018: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:23:34.156 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:34.156 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:33 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:34.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:33 smithi150 ceph-mon[34945]: pgmap v1018: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:23:34.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:34.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:33 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:34.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:33 smithi160 ceph-mon[34696]: pgmap v1018: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:23:34.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:34.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:33 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:34.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 95997 13.43 MB/sec execute 241 sec latency 99.269 ms 2024-09-21T01:23:35.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 96673 13.53 MB/sec execute 242 sec latency 85.542 ms 2024-09-21T01:23:35.889 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:35 smithi039 ceph-mon[30415]: pgmap v1019: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-09-21T01:23:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:23:36.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:35 smithi150 ceph-mon[34945]: pgmap v1019: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-09-21T01:23:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:23:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:35 smithi160 ceph-mon[34696]: pgmap v1019: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-09-21T01:23:36.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:36.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:23:36.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:23:36.819 INFO:tasks.workunit.client.0.smithi039.stdout: 10 97224 13.61 MB/sec execute 243 sec latency 119.282 ms 2024-09-21T01:23:37.820 INFO:tasks.workunit.client.0.smithi039.stdout: 10 97762 13.67 MB/sec execute 244 sec latency 119.570 ms 2024-09-21T01:23:38.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:37 smithi039 ceph-mon[30415]: pgmap v1020: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 854 KiB/s wr, 48 op/s 2024-09-21T01:23:38.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:37 smithi150 ceph-mon[34945]: pgmap v1020: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 854 KiB/s wr, 48 op/s 2024-09-21T01:23:38.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:37 smithi160 ceph-mon[34696]: pgmap v1020: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 854 KiB/s wr, 48 op/s 2024-09-21T01:23:38.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 98368 13.76 MB/sec execute 245 sec latency 92.206 ms 2024-09-21T01:23:39.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 98860 13.78 MB/sec execute 246 sec latency 104.484 ms 2024-09-21T01:23:40.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:39 smithi039 ceph-mon[30415]: pgmap v1021: 129 pgs: 129 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 224 op/s 2024-09-21T01:23:40.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:39 smithi150 ceph-mon[34945]: pgmap v1021: 129 pgs: 129 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 224 op/s 2024-09-21T01:23:40.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:39 smithi160 ceph-mon[34696]: pgmap v1021: 129 pgs: 129 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 224 op/s 2024-09-21T01:23:40.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 99634 13.88 MB/sec execute 247 sec latency 82.149 ms 2024-09-21T01:23:41.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 100404 14.02 MB/sec execute 248 sec latency 117.247 ms 2024-09-21T01:23:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:41 smithi150 ceph-mon[34945]: pgmap v1022: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 241 op/s 2024-09-21T01:23:42.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:41 smithi160 ceph-mon[34696]: pgmap v1022: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 241 op/s 2024-09-21T01:23:42.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:41 smithi039 ceph-mon[30415]: pgmap v1022: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 241 op/s 2024-09-21T01:23:42.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 101082 14.13 MB/sec execute 249 sec latency 82.423 ms 2024-09-21T01:23:43.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 101633 14.16 MB/sec execute 250 sec latency 105.456 ms 2024-09-21T01:23:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:43 smithi150 ceph-mon[34945]: pgmap v1023: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.3 MiB/s wr, 313 op/s 2024-09-21T01:23:44.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:43 smithi160 ceph-mon[34696]: pgmap v1023: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.3 MiB/s wr, 313 op/s 2024-09-21T01:23:44.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:43 smithi039 ceph-mon[30415]: pgmap v1023: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.3 MiB/s wr, 313 op/s 2024-09-21T01:23:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 102197 14.23 MB/sec execute 251 sec latency 124.084 ms 2024-09-21T01:23:45.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 102666 14.28 MB/sec execute 252 sec latency 296.710 ms 2024-09-21T01:23:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:45 smithi150 ceph-mon[34945]: pgmap v1024: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.6 MiB/s wr, 386 op/s 2024-09-21T01:23:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:45 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:23:46.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:45 smithi160 ceph-mon[34696]: pgmap v1024: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.6 MiB/s wr, 386 op/s 2024-09-21T01:23:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:45 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:23:46.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:45 smithi039 ceph-mon[30415]: pgmap v1024: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.6 MiB/s wr, 386 op/s 2024-09-21T01:23:46.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:45 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:23:46.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 103340 14.34 MB/sec execute 253 sec latency 95.059 ms 2024-09-21T01:23:47.821 INFO:tasks.workunit.client.0.smithi039.stdout: 10 104062 14.47 MB/sec execute 254 sec latency 187.560 ms 2024-09-21T01:23:48.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:47 smithi150 ceph-mon[34945]: pgmap v1025: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.3 MiB/s wr, 418 op/s 2024-09-21T01:23:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:47 smithi160 ceph-mon[34696]: pgmap v1025: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.3 MiB/s wr, 418 op/s 2024-09-21T01:23:48.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:47 smithi039 ceph-mon[30415]: pgmap v1025: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.3 MiB/s wr, 418 op/s 2024-09-21T01:23:48.822 INFO:tasks.workunit.client.0.smithi039.stdout: 10 104680 14.54 MB/sec execute 255 sec latency 121.934 ms 2024-09-21T01:23:49.822 INFO:tasks.workunit.client.0.smithi039.stdout: 10 105003 14.55 MB/sec execute 256 sec latency 459.050 ms 2024-09-21T01:23:50.313 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:49 smithi039 ceph-mon[30415]: pgmap v1026: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.4 MiB/s wr, 521 op/s 2024-09-21T01:23:50.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:49 smithi150 ceph-mon[34945]: pgmap v1026: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.4 MiB/s wr, 521 op/s 2024-09-21T01:23:50.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:49 smithi160 ceph-mon[34696]: pgmap v1026: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.4 MiB/s wr, 521 op/s 2024-09-21T01:23:50.822 INFO:tasks.workunit.client.0.smithi039.stdout: 10 105490 14.59 MB/sec execute 257 sec latency 187.125 ms 2024-09-21T01:23:51.214 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:50 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:23:51.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:50 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:23:51.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:50 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:23:51.822 INFO:tasks.workunit.client.0.smithi039.stdout: 10 105989 14.63 MB/sec execute 258 sec latency 244.716 ms 2024-09-21T01:23:52.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:51 smithi150 ceph-mon[34945]: pgmap v1027: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 414 op/s 2024-09-21T01:23:52.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:51 smithi160 ceph-mon[34696]: pgmap v1027: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 414 op/s 2024-09-21T01:23:52.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:51 smithi039 ceph-mon[30415]: pgmap v1027: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 414 op/s 2024-09-21T01:23:52.838 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106521 14.68 MB/sec execute 259 sec latency 130.919 ms 2024-09-21T01:23:53.840 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106537 14.63 MB/sec execute 260 sec latency 1116.570 ms 2024-09-21T01:23:54.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:53 smithi039 ceph-mon[30415]: pgmap v1028: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 424 op/s 2024-09-21T01:23:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:53 smithi150 ceph-mon[34945]: pgmap v1028: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 424 op/s 2024-09-21T01:23:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:53 smithi160 ceph-mon[34696]: pgmap v1028: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 424 op/s 2024-09-21T01:23:54.840 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106539 14.57 MB/sec execute 261 sec latency 2116.639 ms 2024-09-21T01:23:55.190 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:55 smithi039 ceph-mon[30415]: pgmap v1029: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.3 MiB/s wr, 408 op/s 2024-09-21T01:23:55.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:55 smithi150 ceph-mon[34945]: pgmap v1029: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.3 MiB/s wr, 408 op/s 2024-09-21T01:23:55.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:55 smithi160 ceph-mon[34696]: pgmap v1029: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.3 MiB/s wr, 408 op/s 2024-09-21T01:23:55.840 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106540 14.51 MB/sec execute 262 sec latency 3116.695 ms 2024-09-21T01:23:56.840 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106540 14.46 MB/sec execute 263 sec latency 4116.736 ms 2024-09-21T01:23:57.733 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:57 smithi039 ceph-mon[30415]: pgmap v1030: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.1 MiB/s wr, 402 op/s 2024-09-21T01:23:57.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:57 smithi150 ceph-mon[34945]: pgmap v1030: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.1 MiB/s wr, 402 op/s 2024-09-21T01:23:57.840 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106540 14.40 MB/sec execute 264 sec latency 5116.779 ms 2024-09-21T01:23:57.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:57 smithi160 ceph-mon[34696]: pgmap v1030: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.1 MiB/s wr, 402 op/s 2024-09-21T01:23:58.844 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106640 14.38 MB/sec execute 265 sec latency 6120.686 ms 2024-09-21T01:23:59.543 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:23:59 smithi039 ceph-mon[30415]: pgmap v1031: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 11 MiB/s wr, 462 op/s 2024-09-21T01:23:59.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:23:59 smithi150 ceph-mon[34945]: pgmap v1031: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 11 MiB/s wr, 462 op/s 2024-09-21T01:23:59.844 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106640 14.33 MB/sec execute 266 sec latency 7121.575 ms 2024-09-21T01:23:59.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:23:59 smithi160 ceph-mon[34696]: pgmap v1031: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 11 MiB/s wr, 462 op/s 2024-09-21T01:24:00.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106640 14.28 MB/sec execute 267 sec latency 8121.635 ms 2024-09-21T01:24:01.557 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:01 smithi039 ceph-mon[30415]: pgmap v1032: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 10 MiB/s wr, 395 op/s 2024-09-21T01:24:01.784 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:01 smithi160 ceph-mon[34696]: pgmap v1032: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 10 MiB/s wr, 395 op/s 2024-09-21T01:24:01.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:01 smithi150 ceph-mon[34945]: pgmap v1032: 129 pgs: 129 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 10 MiB/s wr, 395 op/s 2024-09-21T01:24:01.844 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106640 14.22 MB/sec execute 268 sec latency 9121.690 ms 2024-09-21T01:24:02.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 14.17 MB/sec execute 269 sec latency 10121.745 ms 2024-09-21T01:24:03.548 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:03 smithi039 ceph-mon[30415]: pgmap v1033: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 11 MiB/s wr, 361 op/s 2024-09-21T01:24:03.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:03 smithi150 ceph-mon[34945]: pgmap v1033: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 11 MiB/s wr, 361 op/s 2024-09-21T01:24:03.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 14.12 MB/sec execute 270 sec latency 11121.796 ms 2024-09-21T01:24:03.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:03 smithi160 ceph-mon[34696]: pgmap v1033: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 11 MiB/s wr, 361 op/s 2024-09-21T01:24:04.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 14.07 MB/sec execute 271 sec latency 12121.847 ms 2024-09-21T01:24:05.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 14.01 MB/sec execute 272 sec latency 13121.900 ms 2024-09-21T01:24:05.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:05 smithi039 ceph-mon[30415]: pgmap v1034: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 384 op/s 2024-09-21T01:24:06.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:05 smithi150 ceph-mon[34945]: pgmap v1034: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 384 op/s 2024-09-21T01:24:06.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:05 smithi160 ceph-mon[34696]: pgmap v1034: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 384 op/s 2024-09-21T01:24:06.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 13.96 MB/sec execute 273 sec latency 14121.952 ms 2024-09-21T01:24:07.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 13.91 MB/sec execute 274 sec latency 15122.006 ms 2024-09-21T01:24:08.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:07 smithi150 ceph-mon[34945]: pgmap v1035: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.9 MiB/s wr, 333 op/s 2024-09-21T01:24:08.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:07 smithi160 ceph-mon[34696]: pgmap v1035: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.9 MiB/s wr, 333 op/s 2024-09-21T01:24:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:07 smithi039 ceph-mon[30415]: pgmap v1035: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.9 MiB/s wr, 333 op/s 2024-09-21T01:24:08.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106641 13.86 MB/sec execute 275 sec latency 16122.055 ms 2024-09-21T01:24:09.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106644 13.81 MB/sec execute 276 sec latency 17122.107 ms 2024-09-21T01:24:09.910 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:09 smithi039 ceph-mon[30415]: pgmap v1036: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.2 MiB/s wr, 297 op/s 2024-09-21T01:24:10.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:09 smithi150 ceph-mon[34945]: pgmap v1036: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.2 MiB/s wr, 297 op/s 2024-09-21T01:24:10.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:09 smithi160 ceph-mon[34696]: pgmap v1036: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.2 MiB/s wr, 297 op/s 2024-09-21T01:24:10.752 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:10 smithi039 ceph-mon[30415]: osdmap e119: 12 total, 12 up, 12 in 2024-09-21T01:24:10.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106673 13.76 MB/sec execute 277 sec latency 18122.167 ms 2024-09-21T01:24:11.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:10 smithi150 ceph-mon[34945]: osdmap e119: 12 total, 12 up, 12 in 2024-09-21T01:24:11.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:10 smithi160 ceph-mon[34696]: osdmap e119: 12 total, 12 up, 12 in 2024-09-21T01:24:11.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 106673 13.71 MB/sec execute 278 sec latency 19122.222 ms 2024-09-21T01:24:11.896 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:11 smithi039 ceph-mon[30415]: pgmap v1038: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 4.5 MiB/s wr, 198 op/s 2024-09-21T01:24:12.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:11 smithi150 ceph-mon[34945]: pgmap v1038: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 4.5 MiB/s wr, 198 op/s 2024-09-21T01:24:12.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:11 smithi160 ceph-mon[34696]: pgmap v1038: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 4.5 MiB/s wr, 198 op/s 2024-09-21T01:24:12.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 107268 13.77 MB/sec execute 279 sec latency 19242.387 ms 2024-09-21T01:24:13.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 107900 13.86 MB/sec execute 280 sec latency 171.255 ms 2024-09-21T01:24:14.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:13 smithi150 ceph-mon[34945]: pgmap v1039: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.7 MiB/s wr, 152 op/s 2024-09-21T01:24:14.097 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:13 smithi039 ceph-mon[30415]: pgmap v1039: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.7 MiB/s wr, 152 op/s 2024-09-21T01:24:14.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:13 smithi160 ceph-mon[34696]: pgmap v1039: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.7 MiB/s wr, 152 op/s 2024-09-21T01:24:14.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 108268 13.88 MB/sec execute 281 sec latency 381.455 ms 2024-09-21T01:24:15.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 108748 13.92 MB/sec execute 282 sec latency 238.023 ms 2024-09-21T01:24:16.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:15 smithi150 ceph-mon[34945]: pgmap v1040: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-09-21T01:24:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:15 smithi160 ceph-mon[34696]: pgmap v1040: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-09-21T01:24:16.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:15 smithi039 ceph-mon[30415]: pgmap v1040: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-09-21T01:24:16.845 INFO:tasks.workunit.client.0.smithi039.stdout: 10 109270 13.96 MB/sec execute 283 sec latency 148.802 ms 2024-09-21T01:24:17.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 109762 14.00 MB/sec execute 284 sec latency 359.676 ms 2024-09-21T01:24:17.984 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:17 smithi039 ceph-mon[30415]: pgmap v1041: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-09-21T01:24:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:17 smithi150 ceph-mon[34945]: pgmap v1041: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-09-21T01:24:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:17 smithi160 ceph-mon[34696]: pgmap v1041: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-09-21T01:24:18.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 110406 14.07 MB/sec execute 285 sec latency 128.741 ms 2024-09-21T01:24:19.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 111189 14.17 MB/sec execute 286 sec latency 87.244 ms 2024-09-21T01:24:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:19 smithi150 ceph-mon[34945]: pgmap v1042: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 265 op/s 2024-09-21T01:24:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:19 smithi150 ceph-mon[34945]: osdmap e120: 12 total, 12 up, 12 in 2024-09-21T01:24:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:19 smithi160 ceph-mon[34696]: pgmap v1042: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 265 op/s 2024-09-21T01:24:20.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:19 smithi160 ceph-mon[34696]: osdmap e120: 12 total, 12 up, 12 in 2024-09-21T01:24:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:19 smithi039 ceph-mon[30415]: pgmap v1042: 129 pgs: 129 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 265 op/s 2024-09-21T01:24:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:19 smithi039 ceph-mon[30415]: osdmap e120: 12 total, 12 up, 12 in 2024-09-21T01:24:20.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 111870 14.25 MB/sec execute 287 sec latency 104.404 ms 2024-09-21T01:24:21.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 112368 14.30 MB/sec execute 288 sec latency 158.454 ms 2024-09-21T01:24:22.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:21 smithi150 ceph-mon[34945]: pgmap v1044: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.6 MiB/s wr, 329 op/s 2024-09-21T01:24:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:21 smithi160 ceph-mon[34696]: pgmap v1044: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.6 MiB/s wr, 329 op/s 2024-09-21T01:24:22.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:21 smithi039 ceph-mon[30415]: pgmap v1044: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.6 MiB/s wr, 329 op/s 2024-09-21T01:24:22.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 112927 14.36 MB/sec execute 289 sec latency 221.330 ms 2024-09-21T01:24:23.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 113523 14.43 MB/sec execute 290 sec latency 129.088 ms 2024-09-21T01:24:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:23 smithi150 ceph-mon[34945]: pgmap v1045: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 6.4 MiB/s wr, 432 op/s 2024-09-21T01:24:24.077 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:23 smithi039 ceph-mon[30415]: pgmap v1045: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 6.4 MiB/s wr, 432 op/s 2024-09-21T01:24:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:23 smithi160 ceph-mon[34696]: pgmap v1045: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 6.4 MiB/s wr, 432 op/s 2024-09-21T01:24:24.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 113975 14.46 MB/sec execute 291 sec latency 270.387 ms 2024-09-21T01:24:25.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 114621 14.52 MB/sec execute 292 sec latency 165.751 ms 2024-09-21T01:24:25.999 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:25 smithi039 ceph-mon[30415]: pgmap v1046: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.9 MiB/s wr, 506 op/s 2024-09-21T01:24:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:25 smithi150 ceph-mon[34945]: pgmap v1046: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.9 MiB/s wr, 506 op/s 2024-09-21T01:24:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:25 smithi160 ceph-mon[34696]: pgmap v1046: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.9 MiB/s wr, 506 op/s 2024-09-21T01:24:26.846 INFO:tasks.workunit.client.0.smithi039.stdout: 10 115213 14.60 MB/sec execute 293 sec latency 181.458 ms 2024-09-21T01:24:27.847 INFO:tasks.workunit.client.0.smithi039.stdout: 10 115771 14.65 MB/sec execute 294 sec latency 166.009 ms 2024-09-21T01:24:27.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:27 smithi039 ceph-mon[30415]: pgmap v1047: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.7 MiB/s wr, 546 op/s 2024-09-21T01:24:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:27 smithi150 ceph-mon[34945]: pgmap v1047: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.7 MiB/s wr, 546 op/s 2024-09-21T01:24:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:27 smithi160 ceph-mon[34696]: pgmap v1047: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.7 MiB/s wr, 546 op/s 2024-09-21T01:24:28.847 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116288 14.67 MB/sec execute 295 sec latency 109.622 ms 2024-09-21T01:24:29.868 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116553 14.68 MB/sec execute 296 sec latency 620.775 ms 2024-09-21T01:24:29.868 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:29 smithi039 ceph-mon[30415]: pgmap v1048: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.0 MiB/s wr, 461 op/s 2024-09-21T01:24:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:29 smithi150 ceph-mon[34945]: pgmap v1048: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.0 MiB/s wr, 461 op/s 2024-09-21T01:24:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:29 smithi160 ceph-mon[34696]: pgmap v1048: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.0 MiB/s wr, 461 op/s 2024-09-21T01:24:30.871 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116576 14.63 MB/sec execute 297 sec latency 976.325 ms 2024-09-21T01:24:31.871 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116633 14.58 MB/sec execute 298 sec latency 1401.355 ms 2024-09-21T01:24:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:31 smithi150 ceph-mon[34945]: pgmap v1049: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 500 op/s 2024-09-21T01:24:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:31 smithi160 ceph-mon[34696]: pgmap v1049: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 500 op/s 2024-09-21T01:24:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:31 smithi039 ceph-mon[30415]: pgmap v1049: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 500 op/s 2024-09-21T01:24:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116633 14.54 MB/sec execute 299 sec latency 2401.431 ms 2024-09-21T01:24:33.871 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116633 14.49 MB/sec execute 300 sec latency 3401.496 ms 2024-09-21T01:24:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:33 smithi150 ceph-mon[34945]: pgmap v1050: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.6 MiB/s wr, 458 op/s 2024-09-21T01:24:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:33 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:24:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:33 smithi160 ceph-mon[34696]: pgmap v1050: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.6 MiB/s wr, 458 op/s 2024-09-21T01:24:34.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:33 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:24:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:33 smithi039 ceph-mon[30415]: pgmap v1050: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.6 MiB/s wr, 458 op/s 2024-09-21T01:24:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:33 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:24:34.872 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116727 14.48 MB/sec execute 301 sec latency 4401.993 ms 2024-09-21T01:24:35.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.43 MB/sec execute 302 sec latency 5342.308 ms 2024-09-21T01:24:36.059 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:35 smithi039 ceph-mon[30415]: pgmap v1051: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.7 MiB/s wr, 424 op/s 2024-09-21T01:24:36.059 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:35 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:35 smithi150 ceph-mon[34945]: pgmap v1051: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.7 MiB/s wr, 424 op/s 2024-09-21T01:24:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:35 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:35 smithi160 ceph-mon[34696]: pgmap v1051: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.7 MiB/s wr, 424 op/s 2024-09-21T01:24:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:35 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:36.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.38 MB/sec execute 303 sec latency 6342.364 ms 2024-09-21T01:24:37.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.33 MB/sec execute 304 sec latency 7342.420 ms 2024-09-21T01:24:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:37 smithi150 ceph-mon[34945]: pgmap v1052: 129 pgs: 129 active+clean; 1.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.9 MiB/s wr, 366 op/s 2024-09-21T01:24:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:24:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:37 smithi160 ceph-mon[34696]: pgmap v1052: 129 pgs: 129 active+clean; 1.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.9 MiB/s wr, 366 op/s 2024-09-21T01:24:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:24:38.174 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:37 smithi039 ceph-mon[30415]: pgmap v1052: 129 pgs: 129 active+clean; 1.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.9 MiB/s wr, 366 op/s 2024-09-21T01:24:38.174 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:38.174 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:38.174 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:24:38.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.28 MB/sec execute 305 sec latency 8342.478 ms 2024-09-21T01:24:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:38 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:24:39.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:38 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:24:39.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:38 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:24:39.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.24 MB/sec execute 306 sec latency 9342.544 ms 2024-09-21T01:24:40.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:39 smithi150 ceph-mon[34945]: pgmap v1053: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 11 MiB/s wr, 395 op/s 2024-09-21T01:24:40.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:39 smithi160 ceph-mon[34696]: pgmap v1053: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 11 MiB/s wr, 395 op/s 2024-09-21T01:24:40.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:39 smithi039 ceph-mon[30415]: pgmap v1053: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 11 MiB/s wr, 395 op/s 2024-09-21T01:24:40.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.19 MB/sec execute 307 sec latency 10342.608 ms 2024-09-21T01:24:41.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.14 MB/sec execute 308 sec latency 11342.672 ms 2024-09-21T01:24:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:41 smithi039 ceph-mon[30415]: pgmap v1054: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 11 MiB/s wr, 357 op/s 2024-09-21T01:24:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:41 smithi150 ceph-mon[34945]: pgmap v1054: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 11 MiB/s wr, 357 op/s 2024-09-21T01:24:42.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:41 smithi160 ceph-mon[34696]: pgmap v1054: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 11 MiB/s wr, 357 op/s 2024-09-21T01:24:42.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116730 14.10 MB/sec execute 309 sec latency 12342.738 ms 2024-09-21T01:24:43.899 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116832 14.08 MB/sec execute 310 sec latency 12799.823 ms 2024-09-21T01:24:44.126 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:43 smithi039 ceph-mon[30415]: pgmap v1055: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.6 MiB/s wr, 279 op/s 2024-09-21T01:24:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:43 smithi150 ceph-mon[34945]: pgmap v1055: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.6 MiB/s wr, 279 op/s 2024-09-21T01:24:44.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:43 smithi160 ceph-mon[34696]: pgmap v1055: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.6 MiB/s wr, 279 op/s 2024-09-21T01:24:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 116906 14.05 MB/sec execute 311 sec latency 9241.057 ms 2024-09-21T01:24:45.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 117224 14.04 MB/sec execute 312 sec latency 9243.490 ms 2024-09-21T01:24:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:45 smithi039 ceph-mon[30415]: pgmap v1056: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 MiB/s wr, 255 op/s 2024-09-21T01:24:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:45 smithi150 ceph-mon[34945]: pgmap v1056: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 MiB/s wr, 255 op/s 2024-09-21T01:24:46.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:45 smithi160 ceph-mon[34696]: pgmap v1056: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 MiB/s wr, 255 op/s 2024-09-21T01:24:46.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 117865 14.08 MB/sec execute 313 sec latency 101.952 ms 2024-09-21T01:24:47.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 118493 14.16 MB/sec execute 314 sec latency 120.640 ms 2024-09-21T01:24:48.252 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:47 smithi039 ceph-mon[30415]: pgmap v1057: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 200 op/s 2024-09-21T01:24:48.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:47 smithi150 ceph-mon[34945]: pgmap v1057: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 200 op/s 2024-09-21T01:24:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:47 smithi160 ceph-mon[34696]: pgmap v1057: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 200 op/s 2024-09-21T01:24:48.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 119261 14.26 MB/sec execute 315 sec latency 136.101 ms 2024-09-21T01:24:49.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 119829 14.32 MB/sec execute 316 sec latency 112.853 ms 2024-09-21T01:24:50.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:49 smithi150 ceph-mon[34945]: pgmap v1058: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.9 MiB/s wr, 241 op/s 2024-09-21T01:24:50.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:49 smithi160 ceph-mon[34696]: pgmap v1058: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.9 MiB/s wr, 241 op/s 2024-09-21T01:24:50.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:49 smithi039 ceph-mon[30415]: pgmap v1058: 129 pgs: 129 active+clean; 1.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.9 MiB/s wr, 241 op/s 2024-09-21T01:24:50.900 INFO:tasks.workunit.client.0.smithi039.stdout: 10 120394 14.37 MB/sec execute 317 sec latency 122.126 ms 2024-09-21T01:24:51.901 INFO:tasks.workunit.client.0.smithi039.stdout: 10 120966 14.42 MB/sec execute 318 sec latency 99.679 ms 2024-09-21T01:24:52.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:51 smithi150 ceph-mon[34945]: pgmap v1059: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-09-21T01:24:52.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:51 smithi160 ceph-mon[34696]: pgmap v1059: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-09-21T01:24:52.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:51 smithi039 ceph-mon[30415]: pgmap v1059: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-09-21T01:24:52.901 INFO:tasks.workunit.client.0.smithi039.stdout: 10 121573 14.44 MB/sec execute 319 sec latency 83.742 ms 2024-09-21T01:24:53.901 INFO:tasks.workunit.client.0.smithi039.stdout: 10 122437 14.58 MB/sec execute 320 sec latency 107.416 ms 2024-09-21T01:24:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:53 smithi150 ceph-mon[34945]: pgmap v1060: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-09-21T01:24:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:53 smithi160 ceph-mon[34696]: pgmap v1060: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-09-21T01:24:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:53 smithi039 ceph-mon[30415]: pgmap v1060: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-09-21T01:24:54.901 INFO:tasks.workunit.client.0.smithi039.stdout: 10 122999 14.63 MB/sec execute 321 sec latency 106.501 ms 2024-09-21T01:24:55.434 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.3 2024-09-21T01:24:55.434 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:24:55.901 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123475 14.64 MB/sec execute 322 sec latency 216.021 ms 2024-09-21T01:24:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:55 smithi150 ceph-mon[34945]: pgmap v1061: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.5 MiB/s wr, 313 op/s 2024-09-21T01:24:56.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:55 smithi160 ceph-mon[34696]: pgmap v1061: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.5 MiB/s wr, 313 op/s 2024-09-21T01:24:56.663 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:55 smithi039 ceph-mon[30415]: pgmap v1061: 129 pgs: 129 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.5 MiB/s wr, 313 op/s 2024-09-21T01:24:56.901 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123483 14.60 MB/sec execute 323 sec latency 1209.401 ms 2024-09-21T01:24:57.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123490 14.55 MB/sec execute 324 sec latency 2100.549 ms 2024-09-21T01:24:57.963 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:24:58.117 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:24:57 smithi039 ceph-mon[30415]: pgmap v1062: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 5.7 MiB/s wr, 319 op/s 2024-09-21T01:24:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:24:57 smithi150 ceph-mon[34945]: pgmap v1062: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 5.7 MiB/s wr, 319 op/s 2024-09-21T01:24:58.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:24:57 smithi160 ceph-mon[34696]: pgmap v1062: 129 pgs: 129 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 5.7 MiB/s wr, 319 op/s 2024-09-21T01:24:58.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123530 14.52 MB/sec execute 325 sec latency 3100.614 ms 2024-09-21T01:24:59.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123530 14.47 MB/sec execute 326 sec latency 4100.670 ms 2024-09-21T01:25:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:00 smithi150 ceph-mon[34945]: pgmap v1063: 129 pgs: 129 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 481 op/s 2024-09-21T01:25:00.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:00 smithi160 ceph-mon[34696]: pgmap v1063: 129 pgs: 129 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 481 op/s 2024-09-21T01:25:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:00 smithi039 ceph-mon[30415]: pgmap v1063: 129 pgs: 129 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 481 op/s 2024-09-21T01:25:00.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123530 14.43 MB/sec execute 327 sec latency 5100.717 ms 2024-09-21T01:25:01.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123530 14.39 MB/sec execute 328 sec latency 6100.764 ms 2024-09-21T01:25:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:02 smithi150 ceph-mon[34945]: pgmap v1064: 129 pgs: 129 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 13 MiB/s wr, 478 op/s 2024-09-21T01:25:02.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:02 smithi160 ceph-mon[34696]: pgmap v1064: 129 pgs: 129 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 13 MiB/s wr, 478 op/s 2024-09-21T01:25:02.360 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:02 smithi039 ceph-mon[30415]: pgmap v1064: 129 pgs: 129 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 13 MiB/s wr, 478 op/s 2024-09-21T01:25:02.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 123729 14.38 MB/sec execute 329 sec latency 6791.458 ms 2024-09-21T01:25:03.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 124212 14.43 MB/sec execute 330 sec latency 245.588 ms 2024-09-21T01:25:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:04 smithi150 ceph-mon[34945]: pgmap v1065: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 471 op/s 2024-09-21T01:25:04.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:04 smithi160 ceph-mon[34696]: pgmap v1065: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 471 op/s 2024-09-21T01:25:04.396 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:04 smithi039 ceph-mon[30415]: pgmap v1065: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 471 op/s 2024-09-21T01:25:04.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 124595 14.42 MB/sec execute 331 sec latency 256.580 ms 2024-09-21T01:25:05.022 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:05 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2280692826' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:25:05.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 -- 172.21.15.39:0/968530467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c114570 msgr2=0x7f8a5c116990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 --2- 172.21.15.39:0/968530467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c114570 0x7f8a5c116990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009be0 tx=0x7f8a5402a6c0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 -- 172.21.15.39:0/968530467 shutdown_connections 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 --2- 172.21.15.39:0/968530467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c114570 0x7f8a5c116990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 --2- 172.21.15.39:0/968530467 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a5c078290 0x7f8a5c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 --2- 172.21.15.39:0/968530467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c077d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 -- 172.21.15.39:0/968530467 >> 172.21.15.39:0/968530467 conn(0x7f8a5c101680 msgr2=0x7f8a5c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 -- 172.21.15.39:0/968530467 shutdown_connections 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.314+0000 7f8a5be00640 1 -- 172.21.15.39:0/968530467 wait complete. 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.315+0000 7f8a5be00640 1 Processor -- start 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 -- start start 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c078290 0x7f8a5c1a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a5c114570 0x7f8a5c1a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c119ef0 con 0x7f8a5c078290 2024-09-21T01:25:05.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c119d70 con 0x7f8a5c077940 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5be00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c11a070 con 0x7f8a5c114570 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c078290 0x7f8a5c1a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c1a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41880/0 (socket says 172.21.15.39:41880) 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5aa00640 1 -- 172.21.15.39:0/2280692826 learned_addr learned my addr 172.21.15.39:0/2280692826 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.316+0000 7f8a5b400640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a5c114570 0x7f8a5c1a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5aa00640 1 -- 172.21.15.39:0/2280692826 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a5c114570 msgr2=0x7f8a5c1a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5aa00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a5c114570 0x7f8a5c1a6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5aa00640 1 -- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c078290 msgr2=0x7f8a5c1a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5aa00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c078290 0x7f8a5c1a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5aa00640 1 -- 172.21.15.39:0/2280692826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c1a6830 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5aa00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c1a08c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00ea60 tx=0x7f8a4c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00cd20 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5c1a6b20 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c0035f0 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c010690 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.317+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5c1ad280 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.318+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a5c077d50 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.321+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8a4c0037a0 con 0x7f8a5c077940 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.322+0000 7f8a60a00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a400b6680 0x7f8a400b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:05.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.322+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9511+0+0 (secure 0 0 0) 0x7f8a4c0e7ff0 con 0x7f8a5c077940 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.322+0000 7f8a5a000640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a400b6680 0x7f8a400b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.323+0000 7f8a5a000640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a400b6680 0x7f8a400b8b90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c07a200 tx=0x7f8a48009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.324+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8a4c056710 con 0x7f8a5c077940 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.701+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a5c1a19f0 con 0x7f8a5c077940 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":60,"btime":"2024-09-21T01:24:37:894131+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":60,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:24:37.122805+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:25:05.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.702+0000 7f8a60a00640 1 -- 172.21.15.39:0/2280692826 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11918 (secure 0 0 0) 0x7f8a4c0ae140 con 0x7f8a5c077940 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 60 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.706+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a400b6680 msgr2=0x7f8a400b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.706+0000 7f8a5be00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a400b6680 0x7f8a400b8b90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c07a200 tx=0x7f8a48009260 comp rx=0 tx=0).stop 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.706+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 msgr2=0x7f8a5c1a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.706+0000 7f8a5be00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c1a08c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00ea60 tx=0x7f8a4c00ee60 comp rx=0 tx=0).stop 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 shutdown_connections 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a5c114570 0x7f8a5c1a6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a5c078290 0x7f8a5c1a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8a400b6680 0x7f8a400b8b90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 --2- 172.21.15.39:0/2280692826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a5c077940 0x7f8a5c1a08c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 >> 172.21.15.39:0/2280692826 conn(0x7f8a5c101680 msgr2=0x7f8a5c1165a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 shutdown_connections 2024-09-21T01:25:05.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:04.707+0000 7f8a5be00640 1 -- 172.21.15.39:0/2280692826 wait complete. 2024-09-21T01:25:05.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:05 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2280692826' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:25:05.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:05 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2280692826' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:25:05.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 125054 14.44 MB/sec execute 332 sec latency 200.819 ms 2024-09-21T01:25:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:06 smithi150 ceph-mon[34945]: pgmap v1066: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 11 MiB/s wr, 424 op/s 2024-09-21T01:25:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:06 smithi150 ceph-mon[34945]: osdmap e121: 12 total, 12 up, 12 in 2024-09-21T01:25:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:06 smithi160 ceph-mon[34696]: pgmap v1066: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 11 MiB/s wr, 424 op/s 2024-09-21T01:25:06.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:06 smithi160 ceph-mon[34696]: osdmap e121: 12 total, 12 up, 12 in 2024-09-21T01:25:06.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:06 smithi039 ceph-mon[30415]: pgmap v1066: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 11 MiB/s wr, 424 op/s 2024-09-21T01:25:06.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:06 smithi039 ceph-mon[30415]: osdmap e121: 12 total, 12 up, 12 in 2024-09-21T01:25:06.903 INFO:tasks.workunit.client.0.smithi039.stdout: 10 125582 14.51 MB/sec execute 333 sec latency 285.605 ms 2024-09-21T01:25:07.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:07 smithi150 ceph-mon[34945]: pgmap v1068: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 11 MiB/s wr, 409 op/s 2024-09-21T01:25:07.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:07 smithi160 ceph-mon[34696]: pgmap v1068: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 11 MiB/s wr, 409 op/s 2024-09-21T01:25:07.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:07 smithi039 ceph-mon[30415]: pgmap v1068: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 11 MiB/s wr, 409 op/s 2024-09-21T01:25:07.902 INFO:tasks.workunit.client.0.smithi039.stdout: 10 126104 14.55 MB/sec execute 334 sec latency 125.970 ms 2024-09-21T01:25:07.974 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0' 2024-09-21T01:25:07.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0 2024-09-21T01:25:08.768 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:25:08.903 INFO:tasks.workunit.client.0.smithi039.stdout: 10 126456 14.56 MB/sec execute 335 sec latency 174.741 ms 2024-09-21T01:25:09.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:09 smithi150 ceph-mon[34945]: pgmap v1069: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.4 MiB/s wr, 357 op/s 2024-09-21T01:25:09.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:09 smithi160 ceph-mon[34696]: pgmap v1069: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.4 MiB/s wr, 357 op/s 2024-09-21T01:25:09.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:09 smithi039 ceph-mon[30415]: pgmap v1069: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.4 MiB/s wr, 357 op/s 2024-09-21T01:25:09.904 INFO:tasks.workunit.client.0.smithi039.stdout: 10 126828 14.58 MB/sec execute 336 sec latency 385.868 ms 2024-09-21T01:25:10.904 INFO:tasks.workunit.client.0.smithi039.stdout: 10 127303 14.63 MB/sec execute 337 sec latency 156.156 ms 2024-09-21T01:25:11.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:11 smithi150 ceph-mon[34945]: pgmap v1070: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.2 MiB/s wr, 352 op/s 2024-09-21T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:11 smithi160 ceph-mon[34696]: pgmap v1070: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.2 MiB/s wr, 352 op/s 2024-09-21T01:25:11.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:11 smithi039 ceph-mon[30415]: pgmap v1070: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.2 MiB/s wr, 352 op/s 2024-09-21T01:25:11.904 INFO:tasks.workunit.client.0.smithi039.stdout: 10 127521 14.61 MB/sec execute 338 sec latency 510.415 ms 2024-09-21T01:25:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.889+0000 7f681b200640 1 -- 172.21.15.39:0/3227467163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6814096f80 msgr2=0x7f6814097390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.889+0000 7f681b200640 1 --2- 172.21.15.39:0/3227467163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6814096f80 0x7f6814097390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f680c009900 tx=0x7f680c02a410 comp rx=0 tx=0).stop 2024-09-21T01:25:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.891+0000 7f681b200640 1 -- 172.21.15.39:0/3227467163 shutdown_connections 2024-09-21T01:25:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.891+0000 7f681b200640 1 --2- 172.21.15.39:0/3227467163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 0x7f68140a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.891+0000 7f681b200640 1 --2- 172.21.15.39:0/3227467163 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f68140978d0 0x7f681409fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.891+0000 7f681b200640 1 --2- 172.21.15.39:0/3227467163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6814096f80 0x7f6814097390 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.891+0000 7f681b200640 1 -- 172.21.15.39:0/3227467163 >> 172.21.15.39:0/3227467163 conn(0x7f681400cd80 msgr2=0x7f681400d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:12.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.892+0000 7f681b200640 1 -- 172.21.15.39:0/3227467163 shutdown_connections 2024-09-21T01:25:12.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.893+0000 7f681b200640 1 -- 172.21.15.39:0/3227467163 wait complete. 2024-09-21T01:25:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.894+0000 7f681b200640 1 Processor -- start 2024-09-21T01:25:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.894+0000 7f681b200640 1 -- start start 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.895+0000 7f681b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6814096f80 0x7f681412cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.895+0000 7f681b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 0x7f681412d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.895+0000 7f681b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 0x7f68141322f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.895+0000 7f681b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f68140a60a0 con 0x7f68140a03e0 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.895+0000 7f681b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68140a5f20 con 0x7f68140978d0 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.895+0000 7f681b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f68140a6220 con 0x7f6814096f80 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.896+0000 7f6819400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 0x7f681412d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.896+0000 7f6819400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 0x7f681412d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43178/0 (socket says 172.21.15.39:43178) 2024-09-21T01:25:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.896+0000 7f6819400640 1 -- 172.21.15.39:0/2745542165 learned_addr learned my addr 172.21.15.39:0/2745542165 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.896+0000 7f681a800640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 0x7f68141322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f6819e00640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6814096f80 0x7f681412cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f6819400640 1 -- 172.21.15.39:0/2745542165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6814096f80 msgr2=0x7f681412cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f6819400640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6814096f80 0x7f681412cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f6819400640 1 -- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 msgr2=0x7f68141322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f6819400640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 0x7f68141322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f6819400640 1 -- 172.21.15.39:0/2745542165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68141329f0 con 0x7f68140978d0 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.897+0000 7f681a800640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 0x7f68141322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:25:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.898+0000 7f6819400640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 0x7f681412d050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f680800e9f0 tx=0x7f680800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.899+0000 7f6812a00640 1 -- 172.21.15.39:0/2745542165 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680800ccc0 con 0x7f68140978d0 2024-09-21T01:25:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.899+0000 7f6812a00640 1 -- 172.21.15.39:0/2745542165 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68080108e0 con 0x7f68140978d0 2024-09-21T01:25:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.899+0000 7f6812a00640 1 -- 172.21.15.39:0/2745542165 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808018860 con 0x7f68140978d0 2024-09-21T01:25:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.899+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6814132ce0 con 0x7f68140978d0 2024-09-21T01:25:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.900+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6814094ba0 con 0x7f68140978d0 2024-09-21T01:25:12.905 INFO:tasks.workunit.client.0.smithi039.stdout: 10 127965 14.63 MB/sec execute 339 sec latency 224.031 ms 2024-09-21T01:25:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.913+0000 7f6812a00640 1 -- 172.21.15.39:0/2745542165 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6808010430 con 0x7f68140978d0 2024-09-21T01:25:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.913+0000 7f6827c98740 1 Processor -- start 2024-09-21T01:25:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.915+0000 7f6827c98740 1 -- start start 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.915+0000 7f6827c98740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b9771c840 0x559b97ac4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.915+0000 7f6827c98740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97ac4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.915+0000 7f6827c98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97ac4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.915+0000 7f6827c98740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559b97a39ca0 con 0x559b9771c840 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.915+0000 7f6827c98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559b97a290d0 con 0x559b976fbb60 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.916+0000 7f6827c98740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b97a2cee0 con 0x559b976f3980 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.916+0000 7f6819400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97ac4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.916+0000 7f6819400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97ac4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43182/0 (socket says 172.21.15.39:43182) 2024-09-21T01:25:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.916+0000 7f6819400640 1 -- 172.21.15.39:0/1373375967 learned_addr learned my addr 172.21.15.39:0/1373375967 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.916+0000 7f6819e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97ac4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.917+0000 7f681a800640 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b9771c840 0x559b97ac4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.917+0000 7f6812a00640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68000b6800 0x7f68000b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.917+0000 7f6812a00640 1 -- 172.21.15.39:0/2745542165 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9599+0+0 (secure 0 0 0) 0x7f68080e7f70 con 0x7f68140978d0 2024-09-21T01:25:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.918+0000 7f681a800640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68000b6800 0x7f68000b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f6819400640 1 -- 172.21.15.39:0/1373375967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 msgr2=0x559b97ac4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f6819400640 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97ac4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f6819400640 1 -- 172.21.15.39:0/1373375967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b9771c840 msgr2=0x559b97ac4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f6819400640 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b9771c840 0x559b97ac4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f6819400640 1 -- 172.21.15.39:0/1373375967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b979db400 con 0x559b976fbb60 2024-09-21T01:25:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f681a800640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68000b6800 0x7f68000b8d10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f681c051230 tx=0x7f681c069720 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.919+0000 7f6819400640 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97ac4f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f680800e9c0 tx=0x7f6808000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2438bbf1809390c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.920+0000 7f6807400640 1 -- 172.21.15.39:0/1373375967 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68080a9100 con 0x559b976fbb60 2024-09-21T01:25:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.920+0000 7f6807400640 1 -- 172.21.15.39:0/1373375967 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6808010720 con 0x559b976fbb60 2024-09-21T01:25:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.920+0000 7f6807400640 1 -- 172.21.15.39:0/1373375967 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68080ea0d0 con 0x559b976fbb60 2024-09-21T01:25:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.924+0000 7f6827c98740 1 -- 172.21.15.39:0/1373375967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 msgr2=0x559b97ac4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.924+0000 7f6827c98740 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97ac4f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f680800e9c0 tx=0x7f6808000bc0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.927+0000 7f6827c98740 1 -- 172.21.15.39:0/1373375967 shutdown_connections 2024-09-21T01:25:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.927+0000 7f6827c98740 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97ac4f80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.927+0000 7f6827c98740 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97ac4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.927+0000 7f6827c98740 1 --2- 172.21.15.39:0/1373375967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b9771c840 0x559b97ac4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.927+0000 7f6827c98740 1 -- 172.21.15.39:0/1373375967 >> 172.21.15.39:0/1373375967 conn(0x559b97710990 msgr2=0x559b97ac23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:12.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.928+0000 7f6827c98740 1 -- 172.21.15.39:0/1373375967 shutdown_connections 2024-09-21T01:25:12.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.929+0000 7f6827c98740 1 -- 172.21.15.39:0/1373375967 wait complete. 2024-09-21T01:25:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.930+0000 7f6827c98740 1 -- start start 2024-09-21T01:25:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.930+0000 7f6827c98740 1 Processor -- start 2024-09-21T01:25:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.932+0000 7f6827c98740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976f3980 0x559b97acfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.932+0000 7f6827c98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97acffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.932+0000 7f6827c98740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b9771c840 0x559b97ad04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.932+0000 7f6827c98740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559b9788a7a0 con 0x559b976f3980 2024-09-21T01:25:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.932+0000 7f6827c98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559b979cea10 con 0x559b976fbb60 2024-09-21T01:25:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.932+0000 7f6827c98740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x559b97ae3d30 con 0x559b9771c840 2024-09-21T01:25:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.933+0000 7f6827c98740 10 client.0 _ll_register_callbacks cb 0x559b979debe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:25:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.933+0000 7f681a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b9771c840 0x559b97ad04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.933+0000 7f681a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b9771c840 0x559b97ad04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:49320/0 (socket says 172.21.15.39:49320) 2024-09-21T01:25:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.933+0000 7f681a800640 1 -- 172.21.15.39:0/4168013214 learned_addr learned my addr 172.21.15.39:0/4168013214 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.934+0000 7f6819e00640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976f3980 0x559b97acfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.934+0000 7f6819400640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97acffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.935+0000 7f6819400640 1 -- 172.21.15.39:0/4168013214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b9771c840 msgr2=0x559b97ad04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.935+0000 7f6819400640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b9771c840 0x559b97ad04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.935+0000 7f6819400640 1 -- 172.21.15.39:0/4168013214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976f3980 msgr2=0x559b97acfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.935+0000 7f6819400640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976f3980 0x559b97acfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.935+0000 7f6819400640 1 -- 172.21.15.39:0/4168013214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b97796b90 con 0x559b976fbb60 2024-09-21T01:25:12.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.938+0000 7f6819400640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97acffb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f680800e9c0 tx=0x7f6808009740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ab57a8ff62f7773 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.938+0000 7f67fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.938+0000 7f67fbe00640 1 -- 172.21.15.39:0/4168013214 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68080a9100 con 0x559b976fbb60 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.938+0000 7f67fbe00640 1 -- 172.21.15.39:0/4168013214 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9599+0+0 (secure 0 0 0) 0x7f68080a92e0 con 0x559b976fbb60 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.938+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b97a3bbf0 con 0x559b976fbb60 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.939+0000 7f67fbe00640 1 -- 172.21.15.39:0/4168013214 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f68080b2470 con 0x559b976fbb60 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.939+0000 7f6827c98740 10 client.26411 fetch_fsmap learned FSMap version 60 2024-09-21T01:25:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.939+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x559b97aef940 con 0x559b976fbb60 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.940+0000 7f67fbe00640 1 -- 172.21.15.39:0/4168013214 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559b97a3bbf0 con 0x559b976fbb60 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f67fbe00640 1 -- 172.21.15.39:0/4168013214 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 60) ==== 8374+0+0 (secure 0 0 0) 0x7f68080ac760 con 0x559b976fbb60 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f6827c98740 10 client.26411 fetch_fsmap finished waiting for FSMap version 60 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f6827c98740 10 client.26411 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f6827c98740 20 client.26411 populate_metadata read hostname 'smithi039' 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f6827c98740 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976db1c0 0x559b97af2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f6827c98740 4 client.26411 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.941+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b97a376a0 con 0x559b976db1c0 2024-09-21T01:25:12.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.942+0000 7f6819e00640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976db1c0 0x559b97af2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.943+0000 7f6819e00640 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976db1c0 0x559b97af2ca0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f09833196e3466a server_cookie=33aadd36de97cf32 in_seq=0 out_seq=0 2024-09-21T01:25:12.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.943+0000 7f67fbe00640 10 client.26411 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:12.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.945+0000 7f67fbe00640 1 -- 172.21.15.39:0/4168013214 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559b97a376a0 con 0x559b976db1c0 2024-09-21T01:25:12.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.945+0000 7f67fbe00640 10 client.26411 handle_command_reply: tid=0 2024-09-21T01:25:12.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.945+0000 7f6827c98740 1 client.26411 shutdown 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 msgr2=0x559b97acffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97acffb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f680800e9c0 tx=0x7f6808009740 comp rx=0 tx=0).stop 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 shutdown_connections 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976db1c0 0x559b97af2ca0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b9771c840 0x559b97ad04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b976fbb60 0x559b97acffb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 --2- 172.21.15.39:0/4168013214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976f3980 0x559b97acfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 >> 172.21.15.39:0/4168013214 conn(0x559b97710990 msgr2=0x559b97ac0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:12.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f67fbe00640 0 client.26411 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 shutdown_connections 2024-09-21T01:25:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.946+0000 7f6827c98740 1 -- 172.21.15.39:0/4168013214 wait complete. 2024-09-21T01:25:12.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.947+0000 7f6827c98740 20 client.26411 trim_cache size 0 max 16384 2024-09-21T01:25:12.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 Processor -- start 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 -- start start 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97accc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97acd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97aec030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6814143ec0 con 0x559b976fbb60 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6814141700 con 0x559b9771c840 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.987+0000 7f6827c98740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f681413f570 con 0x559b976f3980 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.988+0000 7f681a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97aec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.988+0000 7f681a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97aec030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43192/0 (socket says 172.21.15.39:43192) 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.988+0000 7f681a800640 1 -- 172.21.15.39:0/356412348 learned_addr learned my addr 172.21.15.39:0/356412348 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.988+0000 7f681a800640 1 -- 172.21.15.39:0/356412348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 msgr2=0x559b97accc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:25:12.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.988+0000 7f681a800640 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97accc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.988+0000 7f681a800640 1 -- 172.21.15.39:0/356412348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 msgr2=0x559b97acd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:12.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.989+0000 7f681a800640 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97acd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.989+0000 7f681a800640 1 -- 172.21.15.39:0/356412348 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b97ae3d90 con 0x559b9771c840 2024-09-21T01:25:12.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.989+0000 7f681a800640 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97aec030 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f681c02ebd0 tx=0x7f681c06c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=830e0fcf942a1274 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:12.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.990+0000 7f6806a00640 1 -- 172.21.15.39:0/356412348 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c08e7d0 con 0x559b9771c840 2024-09-21T01:25:12.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.990+0000 7f6806a00640 1 -- 172.21.15.39:0/356412348 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f681c08ee20 con 0x559b9771c840 2024-09-21T01:25:12.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.990+0000 7f6806a00640 1 -- 172.21.15.39:0/356412348 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c097e40 con 0x559b9771c840 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.991+0000 7f6827c98740 1 -- 172.21.15.39:0/356412348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 msgr2=0x559b97aec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.991+0000 7f6827c98740 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97aec030 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f681c02ebd0 tx=0x7f681c06c000 comp rx=0 tx=0).stop 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 -- 172.21.15.39:0/356412348 shutdown_connections 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97aec030 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97acd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 --2- 172.21.15.39:0/356412348 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97accc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:12.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 -- 172.21.15.39:0/356412348 >> 172.21.15.39:0/356412348 conn(0x559b97710990 msgr2=0x559b97ad0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:12.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 -- 172.21.15.39:0/356412348 shutdown_connections 2024-09-21T01:25:12.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.992+0000 7f6827c98740 1 -- 172.21.15.39:0/356412348 wait complete. 2024-09-21T01:25:12.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.994+0000 7f6827c98740 1 -- start start 2024-09-21T01:25:12.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.994+0000 7f6827c98740 1 Processor -- start 2024-09-21T01:25:12.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97af87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97af6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97af7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:12.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f681413fbc0 con 0x559b976fbb60 2024-09-21T01:25:12.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6814141030 con 0x559b9771c840 2024-09-21T01:25:12.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6814140290 con 0x559b976f3980 2024-09-21T01:25:12.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.996+0000 7f6827c98740 10 client.0 _ll_register_callbacks cb 0x7f680c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:25:12.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.997+0000 7f6819400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97af6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.997+0000 7f681a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97af7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:12.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.997+0000 7f6819400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97af6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37100/0 (socket says 172.21.15.39:37100) 2024-09-21T01:25:12.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.997+0000 7f6819400640 1 -- 172.21.15.39:0/1145935206 learned_addr learned my addr 172.21.15.39:0/1145935206 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.997+0000 7f6819e00640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97af87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.998+0000 7f681a800640 1 -- 172.21.15.39:0/1145935206 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 msgr2=0x559b97af87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.998+0000 7f681a800640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97af87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.998+0000 7f681a800640 1 -- 172.21.15.39:0/1145935206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 msgr2=0x559b97af6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.998+0000 7f681a800640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97af6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.998+0000 7f681a800640 1 -- 172.21.15.39:0/1145935206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b97af7800 con 0x559b9771c840 2024-09-21T01:25:13.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.998+0000 7f6819e00640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97af87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:25:13.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.999+0000 7f681a800640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97af7240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f681c052860 tx=0x7f681c08ebc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e93912589ee65872 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:13.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:12.999+0000 7f67fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:25:13.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.000+0000 7f67fbe00640 1 -- 172.21.15.39:0/1145935206 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c0a7060 con 0x559b9771c840 2024-09-21T01:25:13.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.000+0000 7f67fbe00640 1 -- 172.21.15.39:0/1145935206 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9599+0+0 (secure 0 0 0) 0x7f681c09f070 con 0x559b9771c840 2024-09-21T01:25:13.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.000+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f681c08ee20 con 0x559b9771c840 2024-09-21T01:25:13.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.000+0000 7f67fbe00640 1 -- 172.21.15.39:0/1145935206 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f681c0a7060 con 0x559b9771c840 2024-09-21T01:25:13.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.001+0000 7f6827c98740 10 client.26417 fetch_fsmap learned FSMap version 60 2024-09-21T01:25:13.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.001+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x559b97b5e140 con 0x559b9771c840 2024-09-21T01:25:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.001+0000 7f67fbe00640 1 -- 172.21.15.39:0/1145935206 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f681c08ee20 con 0x559b9771c840 2024-09-21T01:25:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f67fbe00640 1 -- 172.21.15.39:0/1145935206 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 60) ==== 8374+0+0 (secure 0 0 0) 0x7f681c095020 con 0x559b9771c840 2024-09-21T01:25:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f6827c98740 10 client.26417 fetch_fsmap finished waiting for FSMap version 60 2024-09-21T01:25:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f6827c98740 10 client.26417 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:25:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f6827c98740 20 client.26417 populate_metadata read hostname 'smithi039' 2024-09-21T01:25:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f6827c98740 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976b0570 0x559b97b60860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f6827c98740 4 client.26417 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}] 2024-09-21T01:25:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.002+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}) -- 0x559b97a34ad0 con 0x559b976b0570 2024-09-21T01:25:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.003+0000 7f6819e00640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976b0570 0x559b97b60860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.004+0000 7f6819e00640 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976b0570 0x559b97b60860 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac4d1155be288124 server_cookie=62896e6ccf3d1e2e in_seq=0 out_seq=0 2024-09-21T01:25:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.004+0000 7f67fbe00640 10 client.26417 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.678+0000 7f67fbe00640 1 -- 172.21.15.39:0/1145935206 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559b97a34ad0 con 0x559b976b0570 2024-09-21T01:25:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.678+0000 7f67fbe00640 10 client.26417 handle_command_reply: tid=0 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.678+0000 7f6827c98740 1 client.26417 shutdown 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 msgr2=0x559b97af7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97af7240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f681c052860 tx=0x7f681c08ebc0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 shutdown_connections 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x559b976b0570 0x559b97b60860 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559b9771c840 0x559b97af7240 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b976fbb60 0x559b97af6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 --2- 172.21.15.39:0/1145935206 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x559b976f3980 0x559b97af87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 >> 172.21.15.39:0/1145935206 conn(0x559b97710990 msgr2=0x559b97aec9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.679+0000 7f67fbe00640 0 client.26417 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.680+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 shutdown_connections 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.680+0000 7f6827c98740 1 -- 172.21.15.39:0/1145935206 wait complete. 2024-09-21T01:25:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.680+0000 7f6827c98740 20 client.26417 trim_cache size 0 max 16384 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 60, 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 82, 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "42f0107a": { 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 82, 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:25:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 142.3, 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:13.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68000b6800 msgr2=0x7f68000b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68000b6800 0x7f68000b8d10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f681c051230 tx=0x7f681c069720 comp rx=0 tx=0).stop 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 msgr2=0x7f681412d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 0x7f681412d050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f680800e9f0 tx=0x7f680800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 shutdown_connections 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68140a03e0 0x7f68141322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f68000b6800 0x7f68000b8d10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140978d0 0x7f681412d050 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 --2- 172.21.15.39:0/2745542165 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6814096f80 0x7f681412cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.682+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 >> 172.21.15.39:0/2745542165 conn(0x7f681400cd80 msgr2=0x7f6814099f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.683+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 shutdown_connections 2024-09-21T01:25:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:13.683+0000 7f681b200640 1 -- 172.21.15.39:0/2745542165 wait complete. 2024-09-21T01:25:13.814 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:13 smithi150 ceph-mon[34945]: pgmap v1071: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 9.1 MiB/s wr, 355 op/s 2024-09-21T01:25:13.856 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:13 smithi160 ceph-mon[34696]: pgmap v1071: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 9.1 MiB/s wr, 355 op/s 2024-09-21T01:25:13.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:13 smithi039 ceph-mon[30415]: pgmap v1071: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 9.1 MiB/s wr, 355 op/s 2024-09-21T01:25:13.905 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.59 MB/sec execute 340 sec latency 968.239 ms 2024-09-21T01:25:14.842 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '42f0107a', quiesce took 19.4 seconds. Will release after: 2.900000000000002 2024-09-21T01:25:14.906 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.54 MB/sec execute 341 sec latency 1968.318 ms 2024-09-21T01:25:15.906 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.50 MB/sec execute 342 sec latency 2968.415 ms 2024-09-21T01:25:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:15 smithi150 ceph-mon[34945]: pgmap v1072: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 427 op/s 2024-09-21T01:25:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:15 smithi150 ceph-mon[34945]: osdmap e122: 12 total, 12 up, 12 in 2024-09-21T01:25:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:15 smithi160 ceph-mon[34696]: pgmap v1072: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 427 op/s 2024-09-21T01:25:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:15 smithi160 ceph-mon[34696]: osdmap e122: 12 total, 12 up, 12 in 2024-09-21T01:25:16.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:15 smithi039 ceph-mon[30415]: pgmap v1072: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 427 op/s 2024-09-21T01:25:16.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:15 smithi039 ceph-mon[30415]: osdmap e122: 12 total, 12 up, 12 in 2024-09-21T01:25:16.906 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.46 MB/sec execute 343 sec latency 3968.875 ms 2024-09-21T01:25:17.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:25:17.906 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.42 MB/sec execute 344 sec latency 4968.958 ms 2024-09-21T01:25:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:17 smithi150 ceph-mon[34945]: pgmap v1074: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 423 op/s 2024-09-21T01:25:18.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:17 smithi160 ceph-mon[34696]: pgmap v1074: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 423 op/s 2024-09-21T01:25:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:17 smithi039 ceph-mon[30415]: pgmap v1074: 129 pgs: 129 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 423 op/s 2024-09-21T01:25:18.167 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:25:18.906 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.38 MB/sec execute 345 sec latency 5969.110 ms 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.785+0000 7f99b2000640 1 -- 172.21.15.39:0/669531850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102440 msgr2=0x7f99ac102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.785+0000 7f99b2000640 1 --2- 172.21.15.39:0/669531850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102440 0x7f99ac102850 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f99a0009900 tx=0x7f99a002a480 comp rx=0 tx=0).stop 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.785+0000 7f99b2000640 1 -- 172.21.15.39:0/669531850 shutdown_connections 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.786+0000 7f99b2000640 1 --2- 172.21.15.39:0/669531850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ac10fff0 0x7f99ac112410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.786+0000 7f99b2000640 1 --2- 172.21.15.39:0/669531850 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f99ac102d90 0x7f99ac10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.786+0000 7f99b2000640 1 --2- 172.21.15.39:0/669531850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102440 0x7f99ac102850 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.786+0000 7f99b2000640 1 -- 172.21.15.39:0/669531850 >> 172.21.15.39:0/669531850 conn(0x7f99ac0fe2c0 msgr2=0x7f99ac100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.786+0000 7f99b2000640 1 -- 172.21.15.39:0/669531850 shutdown_connections 2024-09-21T01:25:19.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.786+0000 7f99b2000640 1 -- 172.21.15.39:0/669531850 wait complete. 2024-09-21T01:25:19.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 Processor -- start 2024-09-21T01:25:19.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 -- start start 2024-09-21T01:25:19.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f99ac102440 0x7f99ac19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:19.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 0x7f99ac19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:19.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ac10fff0 0x7f99ac1a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:19.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac115ae0 con 0x7f99ac102d90 2024-09-21T01:25:19.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac115960 con 0x7f99ac10fff0 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.787+0000 7f99b2000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac115c60 con 0x7f99ac102440 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 0x7f99ac19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99b0c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f99ac102440 0x7f99ac19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 0x7f99ac19ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37114/0 (socket says 172.21.15.39:37114) 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 -- 172.21.15.39:0/3662436146 learned_addr learned my addr 172.21.15.39:0/3662436146 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99b1600640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ac10fff0 0x7f99ac1a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 -- 172.21.15.39:0/3662436146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f99ac102440 msgr2=0x7f99ac19c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f99ac102440 0x7f99ac19c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 -- 172.21.15.39:0/3662436146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ac10fff0 msgr2=0x7f99ac1a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ac10fff0 0x7f99ac1a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 -- 172.21.15.39:0/3662436146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ac1a2440 con 0x7f99ac102d90 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.788+0000 7f99abe00640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 0x7f99ac19ca70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f999800e9f0 tx=0x7f999800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.789+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999800ccc0 con 0x7f99ac102d90 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.789+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f999800ce60 con 0x7f99ac102d90 2024-09-21T01:25:19.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.789+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ac1a2730 con 0x7f99ac102d90 2024-09-21T01:25:19.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.789+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998010430 con 0x7f99ac102d90 2024-09-21T01:25:19.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.789+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ac1a8e90 con 0x7f99ac102d90 2024-09-21T01:25:19.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.790+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ac102850 con 0x7f99ac102d90 2024-09-21T01:25:19.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.793+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f99980046c0 con 0x7f99ac102d90 2024-09-21T01:25:19.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.795+0000 7f99a9600640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99940b6680 0x7f99940b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:19.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.795+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9511+0+0 (secure 0 0 0) 0x7f99980e8110 con 0x7f99ac102d90 2024-09-21T01:25:19.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.795+0000 7f99b0c00640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99940b6680 0x7f99940b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:19.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.796+0000 7f99b0c00640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99940b6680 0x7f99940b8b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f99a0005ab0 tx=0x7f99a0005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:19.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:19.796+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9998056710 con 0x7f99ac102d90 2024-09-21T01:25:19.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:19 smithi039 ceph-mon[30415]: pgmap v1075: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 8.7 MiB/s wr, 324 op/s 2024-09-21T01:25:19.906 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.33 MB/sec execute 346 sec latency 6969.174 ms 2024-09-21T01:25:20.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.035+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99ac19d600 con 0x7f99ac102d90 2024-09-21T01:25:20.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.036+0000 7f99a9600640 1 -- 172.21.15.39:0/3662436146 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11918 (secure 0 0 0) 0x7f99ac19d600 con 0x7f99ac102d90 2024-09-21T01:25:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:25:20.038 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":60,"btime":"2024-09-21T01:24:37:894131+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":60,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:24:37.122805+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:25:20.038 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 60 2024-09-21T01:25:20.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99940b6680 msgr2=0x7f99940b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:20.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99940b6680 0x7f99940b8b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f99a0005ab0 tx=0x7f99a0005a20 comp rx=0 tx=0).stop 2024-09-21T01:25:20.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 msgr2=0x7f99ac19ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:20.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 0x7f99ac19ca70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f999800e9f0 tx=0x7f999800edf0 comp rx=0 tx=0).stop 2024-09-21T01:25:20.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 shutdown_connections 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f99940b6680 0x7f99940b8b90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ac10fff0 0x7f99ac1a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ac102d90 0x7f99ac19ca70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 --2- 172.21.15.39:0/3662436146 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f99ac102440 0x7f99ac19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 >> 172.21.15.39:0/3662436146 conn(0x7f99ac0fe2c0 msgr2=0x7f99ac1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 shutdown_connections 2024-09-21T01:25:20.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:20.039+0000 7f99b2000640 1 -- 172.21.15.39:0/3662436146 wait complete. 2024-09-21T01:25:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:19 smithi150 ceph-mon[34945]: pgmap v1075: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 8.7 MiB/s wr, 324 op/s 2024-09-21T01:25:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:19 smithi160 ceph-mon[34696]: pgmap v1075: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 8.7 MiB/s wr, 324 op/s 2024-09-21T01:25:20.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.29 MB/sec execute 347 sec latency 7969.251 ms 2024-09-21T01:25:21.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:20 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3662436146' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:25:21.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:20 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3662436146' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:25:21.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:20 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3662436146' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:25:21.310 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 42f0107a --release --await' 2024-09-21T01:25:21.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 42f0107a --release --await 2024-09-21T01:25:21.753 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:25:21.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128025 14.25 MB/sec execute 348 sec latency 8969.319 ms 2024-09-21T01:25:22.008 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:21 smithi160 ceph-mon[34696]: pgmap v1076: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.6 MiB/s wr, 254 op/s 2024-09-21T01:25:22.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:21 smithi039 ceph-mon[30415]: pgmap v1076: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.6 MiB/s wr, 254 op/s 2024-09-21T01:25:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:21 smithi150 ceph-mon[34945]: pgmap v1076: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.6 MiB/s wr, 254 op/s 2024-09-21T01:25:22.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 -- 172.21.15.39:0/574878756 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 msgr2=0x7f0fc8116750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 --2- 172.21.15.39:0/574878756 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc8116750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009ca0 tx=0x7f0fc002a8b0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 -- 172.21.15.39:0/574878756 shutdown_connections 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 --2- 172.21.15.39:0/574878756 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc8116750 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 --2- 172.21.15.39:0/574878756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fc8075dd0 0x7f0fc80761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 --2- 172.21.15.39:0/574878756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc8077380 0x7f0fc80757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.563+0000 7f0fc7200640 1 -- 172.21.15.39:0/574878756 >> 172.21.15.39:0/574878756 conn(0x7f0fc80ff440 msgr2=0x7f0fc8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.564+0000 7f0fc7200640 1 -- 172.21.15.39:0/574878756 shutdown_connections 2024-09-21T01:25:22.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.564+0000 7f0fc7200640 1 -- 172.21.15.39:0/574878756 wait complete. 2024-09-21T01:25:22.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 Processor -- start 2024-09-21T01:25:22.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 -- start start 2024-09-21T01:25:22.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fc8075dd0 0x7f0fc81a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc8077380 0x7f0fc81a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc81a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8119f10 con 0x7f0fc8075dd0 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8119d90 con 0x7f0fc8077380 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.565+0000 7f0fc7200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc811a090 con 0x7f0fc8114280 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc81a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc8077380 0x7f0fc81a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc81a6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:49386/0 (socket says 172.21.15.39:49386) 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 -- 172.21.15.39:0/775719709 learned_addr learned my addr 172.21.15.39:0/775719709 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fc8075dd0 0x7f0fc81a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 -- 172.21.15.39:0/775719709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc8077380 msgr2=0x7f0fc81a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc8077380 0x7f0fc81a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 -- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fc8075dd0 msgr2=0x7f0fc81a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fc8075dd0 0x7f0fc81a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.566+0000 7f0fc6800640 1 -- 172.21.15.39:0/775719709 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc81a6790 con 0x7f0fc8114280 2024-09-21T01:25:22.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.567+0000 7f0fc6800640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc81a6060 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009430 tx=0x7f0fc0030da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:22.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.567+0000 7f0fbea00640 1 -- 172.21.15.39:0/775719709 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0004040 con 0x7f0fc8114280 2024-09-21T01:25:22.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.567+0000 7f0fbea00640 1 -- 172.21.15.39:0/775719709 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fc0040070 con 0x7f0fc8114280 2024-09-21T01:25:22.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.567+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc81a6a20 con 0x7f0fc8114280 2024-09-21T01:25:22.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.567+0000 7f0fbea00640 1 -- 172.21.15.39:0/775719709 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0045b60 con 0x7f0fc8114280 2024-09-21T01:25:22.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.567+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc81ad1e0 con 0x7f0fc8114280 2024-09-21T01:25:22.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.570+0000 7f0fd4009740 1 Processor -- start 2024-09-21T01:25:22.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.571+0000 7f0fbea00640 1 -- 172.21.15.39:0/775719709 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f0fc00041e0 con 0x7f0fc8114280 2024-09-21T01:25:22.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.571+0000 7f0fd4009740 1 -- start start 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de1b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de1b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fd4009740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55676dd92290 con 0x55676da74520 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fd4009740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55676dd816c0 con 0x55676da53840 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fd4009740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55676dd854d0 con 0x55676da4b660 2024-09-21T01:25:22.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de1b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de1b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37136/0 (socket says 172.21.15.39:37136) 2024-09-21T01:25:22.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fbfe00640 1 -- 172.21.15.39:0/2484483804 learned_addr learned my addr 172.21.15.39:0/2484483804 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:22.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de1b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fc5e00640 1 -- 172.21.15.39:0/2484483804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 msgr2=0x55676de1bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fc5e00640 1 -- 172.21.15.39:0/2484483804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 msgr2=0x55676de1b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de1b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.572+0000 7f0fc5e00640 1 -- 172.21.15.39:0/2484483804 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55676dd0e460 con 0x55676da4b660 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fbea00640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0fac0b6680 0x7f0fac0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de1b880 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800bbf0 tx=0x7f0fb80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420fcb11ad7bf1eb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fbea00640 1 -- 172.21.15.39:0/775719709 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9511+0+0 (secure 0 0 0) 0x7f0fc0108e70 con 0x7f0fc8114280 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0fac0b6680 0x7f0fac0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fb7400640 1 -- 172.21.15.39:0/2484483804 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8004030 con 0x55676da4b660 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fb7400640 1 -- 172.21.15.39:0/2484483804 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb80035f0 con 0x55676da4b660 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.573+0000 7f0fb7400640 1 -- 172.21.15.39:0/2484483804 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8011680 con 0x55676da4b660 2024-09-21T01:25:22.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 -- 172.21.15.39:0/2484483804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 msgr2=0x55676de1b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0fac0b6680 0x7f0fac0b8b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009fd0 tx=0x7f0fb0005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de1b880 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800bbf0 tx=0x7f0fb80107a0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 -- 172.21.15.39:0/2484483804 shutdown_connections 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de1b880 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2484483804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de1b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.574+0000 7f0fd4009740 1 -- 172.21.15.39:0/2484483804 >> 172.21.15.39:0/2484483804 conn(0x55676da68910 msgr2=0x55676de19230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.575+0000 7f0fd4009740 1 -- 172.21.15.39:0/2484483804 shutdown_connections 2024-09-21T01:25:22.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.575+0000 7f0fd4009740 1 -- 172.21.15.39:0/2484483804 wait complete. 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.575+0000 7f0fd4009740 1 -- start start 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.575+0000 7f0fd4009740 1 Processor -- start 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 0x55676de26ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da53840 0x55676de273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da74520 0x55676de32e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55676dbe2bb0 con 0x55676da53840 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55676dd1c6c0 con 0x55676da4b660 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55676de29b20 con 0x55676da74520 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fd4009740 10 client.0 _ll_register_callbacks cb 0x55676dd37490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fc5e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 0x55676de26ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fc5e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 0x55676de26ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:43246/0 (socket says 172.21.15.39:43246) 2024-09-21T01:25:22.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fc6800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da74520 0x55676de32e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da53840 0x55676de273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.576+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1403828444 learned_addr learned my addr 172.21.15.39:0/1403828444 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1403828444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da74520 msgr2=0x55676de32e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da74520 0x55676de32e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1403828444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da53840 msgr2=0x55676de273e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da53840 0x55676de273e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1403828444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55676dadfb00 con 0x55676da4b660 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc6800640 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da74520 0x55676de32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 0x55676de26ea0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8003f20 tx=0x7f0fb8003f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231404fd6258961 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.577+0000 7f0fabe00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fabe00640 1 -- 172.21.15.39:0/1403828444 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb802a060 con 0x55676da4b660 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fb80035f0 con 0x55676da4b660 2024-09-21T01:25:22.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fabe00640 1 -- 172.21.15.39:0/1403828444 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9511+0+0 (secure 0 0 0) 0x7f0fb80037a0 con 0x55676da4b660 2024-09-21T01:25:22.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fabe00640 1 -- 172.21.15.39:0/1403828444 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f0fb8031050 con 0x55676da4b660 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fd4009740 10 client.26435 fetch_fsmap learned FSMap version 60 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55676db56960 con 0x55676da4b660 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fabe00640 1 -- 172.21.15.39:0/1403828444 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb80035f0 con 0x55676da4b660 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.578+0000 7f0fabe00640 1 -- 172.21.15.39:0/1403828444 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 60) ==== 8374+0+0 (secure 0 0 0) 0x7f0fb80047e0 con 0x55676da4b660 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fd4009740 10 client.26435 fetch_fsmap finished waiting for FSMap version 60 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fd4009740 10 client.26435 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fd4009740 20 client.26435 populate_metadata read hostname 'smithi039' 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da32ea0 0x55676de38ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fd4009740 4 client.26435 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55676dd941e0 con 0x55676da32ea0 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.579+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da32ea0 0x55676de38ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.580+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da32ea0 0x55676de38ca0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7660beb73990c937 server_cookie=1b79b5e939456fd5 in_seq=0 out_seq=0 2024-09-21T01:25:22.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.580+0000 7f0fabe00640 10 client.26435 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.580+0000 7f0fabe00640 1 -- 172.21.15.39:0/1403828444 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55676dd941e0 con 0x55676da32ea0 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.580+0000 7f0fabe00640 10 client.26435 handle_command_reply: tid=0 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.580+0000 7f0fd4009740 1 client.26435 shutdown 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 msgr2=0x55676de26ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 0x55676de26ea0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8003f20 tx=0x7f0fb8003f80 comp rx=0 tx=0).stop 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 shutdown_connections 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da32ea0 0x55676de38ca0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da74520 0x55676de32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da53840 0x55676de273e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fabe00640 0 client.26435 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1403828444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da4b660 0x55676de26ea0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 >> 172.21.15.39:0/1403828444 conn(0x55676da68910 msgr2=0x55676de1e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 shutdown_connections 2024-09-21T01:25:22.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 1 -- 172.21.15.39:0/1403828444 wait complete. 2024-09-21T01:25:22.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.581+0000 7f0fd4009740 20 client.26435 trim_cache size 0 max 16384 2024-09-21T01:25:22.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 Processor -- start 2024-09-21T01:25:22.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 -- start start 2024-09-21T01:25:22.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de22080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de20420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81b9c50 con 0x55676da74520 2024-09-21T01:25:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81b7490 con 0x55676da53840 2024-09-21T01:25:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fd4009740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81b5300 con 0x55676da4b660 2024-09-21T01:25:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.600+0000 7f0fc6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de20420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de20420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37142/0 (socket says 172.21.15.39:37142) 2024-09-21T01:25:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 -- 172.21.15.39:0/2103603772 learned_addr learned my addr 172.21.15.39:0/2103603772 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de22080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 -- 172.21.15.39:0/2103603772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 msgr2=0x55676de22080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de22080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 -- 172.21.15.39:0/2103603772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 msgr2=0x55676de1eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 -- 172.21.15.39:0/2103603772 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55676de1da30 con 0x55676da74520 2024-09-21T01:25:22.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de22080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:25:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:25:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.601+0000 7f0fc6800640 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de20420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0fc01061e0 tx=0x7f0fc00ce1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294e94eba6ec3c3e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.602+0000 7f0fb6a00640 1 -- 172.21.15.39:0/2103603772 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00d3070 con 0x55676da74520 2024-09-21T01:25:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.602+0000 7f0fb6a00640 1 -- 172.21.15.39:0/2103603772 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fc0036040 con 0x55676da74520 2024-09-21T01:25:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.602+0000 7f0fb6a00640 1 -- 172.21.15.39:0/2103603772 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00cfbb0 con 0x55676da74520 2024-09-21T01:25:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 -- 172.21.15.39:0/2103603772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 msgr2=0x55676de20420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de20420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0fc01061e0 tx=0x7f0fc00ce1d0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 -- 172.21.15.39:0/2103603772 shutdown_connections 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676de20420 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de1eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 --2- 172.21.15.39:0/2103603772 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de22080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 -- 172.21.15.39:0/2103603772 >> 172.21.15.39:0/2103603772 conn(0x55676da68910 msgr2=0x55676de1cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 -- 172.21.15.39:0/2103603772 shutdown_connections 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.603+0000 7f0fd4009740 1 -- 172.21.15.39:0/2103603772 wait complete. 2024-09-21T01:25:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.604+0000 7f0fd4009740 1 -- start start 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.604+0000 7f0fd4009740 1 Processor -- start 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de3a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de3a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676dea2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81b5950 con 0x55676da74520 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81b6dc0 con 0x55676da53840 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81b6020 con 0x55676da4b660 2024-09-21T01:25:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fd4009740 10 client.0 _ll_register_callbacks cb 0x7f0fb0009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:25:22.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de3a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676dea2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de3a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:49414/0 (socket says 172.21.15.39:49414) 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1338977452 learned_addr learned my addr 172.21.15.39:0/1338977452 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de3a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 msgr2=0x55676de3a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de3a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1338977452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 msgr2=0x55676dea2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676dea2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fc5e00640 1 -- 172.21.15.39:0/1338977452 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55676dea2960 con 0x55676da4b660 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.605+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de3a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.606+0000 7f0fc5e00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de3a430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8010ca0 tx=0x7f0fb8005cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6324ed2f8e0d8c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.606+0000 7f0fabe00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.606+0000 7f0fabe00640 1 -- 172.21.15.39:0/1338977452 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8039070 con 0x55676da4b660 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.606+0000 7f0fabe00640 1 -- 172.21.15.39:0/1338977452 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9511+0+0 (secure 0 0 0) 0x7f0fb80037f0 con 0x55676da4b660 2024-09-21T01:25:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.606+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fc0036040 con 0x55676da4b660 2024-09-21T01:25:22.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fabe00640 1 -- 172.21.15.39:0/1338977452 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f0fb8045050 con 0x55676da4b660 2024-09-21T01:25:22.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 10 client.26419 fetch_fsmap learned FSMap version 60 2024-09-21T01:25:22.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55676dea3bf0 con 0x55676da4b660 2024-09-21T01:25:22.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fabe00640 1 -- 172.21.15.39:0/1338977452 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fc0036040 con 0x55676da4b660 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fabe00640 1 -- 172.21.15.39:0/1338977452 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 60) ==== 8374+0+0 (secure 0 0 0) 0x7f0fb8011750 con 0x55676da4b660 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 10 client.26419 fetch_fsmap finished waiting for FSMap version 60 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 10 client.26419 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 20 client.26419 populate_metadata read hostname 'smithi039' 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da08250 0x55676dea62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 4 client.26419 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "42f0107a", "release": true, "await": true}] 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.607+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "42f0107a", "release": true, "await": true}) -- 0x55676dd8fc90 con 0x55676da08250 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.608+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da08250 0x55676dea62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.608+0000 7f0fbfe00640 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da08250 0x55676dea62f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c37098fe1cd98c62 server_cookie=af6d20e08d612387 in_seq=0 out_seq=0 2024-09-21T01:25:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.608+0000 7f0fabe00640 10 client.26419 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fabe00640 1 -- 172.21.15.39:0/1338977452 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55676dd8fc90 con 0x55676da08250 2024-09-21T01:25:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fabe00640 10 client.26419 handle_command_reply: tid=0 2024-09-21T01:25:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 client.26419 shutdown 2024-09-21T01:25:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 msgr2=0x55676de3a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de3a430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8010ca0 tx=0x7f0fb8005cc0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 shutdown_connections 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55676da08250 0x55676dea62f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55676da74520 0x55676dea2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55676da53840 0x55676de3a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 --2- 172.21.15.39:0/1338977452 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55676da4b660 0x55676de3a430 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 >> 172.21.15.39:0/1338977452 conn(0x55676da68910 msgr2=0x55676de1cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fabe00640 0 client.26419 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 shutdown_connections 2024-09-21T01:25:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.637+0000 7f0fd4009740 1 -- 172.21.15.39:0/1338977452 wait complete. 2024-09-21T01:25:22.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fd4009740 20 client.26419 trim_cache size 0 max 16384 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 60, 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 84, 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "42f0107a": { 2024-09-21T01:25:22.648 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 84, 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 142.3, 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:22.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0fac0b6680 msgr2=0x7f0fac0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0fac0b6680 0x7f0fac0b8b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009fd0 tx=0x7f0fb0005850 comp rx=0 tx=0).stop 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 msgr2=0x7f0fc81a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc81a6060 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009430 tx=0x7f0fc0030da0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 shutdown_connections 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0fc8114280 0x7f0fc81a6060 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f0fac0b6680 0x7f0fac0b8b90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc8077380 0x7f0fc81a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 --2- 172.21.15.39:0/775719709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fc8075dd0 0x7f0fc81a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 >> 172.21.15.39:0/775719709 conn(0x7f0fc80ff440 msgr2=0x7f0fc8110180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 shutdown_connections 2024-09-21T01:25:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:25:22.647+0000 7f0fc7200640 1 -- 172.21.15.39:0/775719709 wait complete. 2024-09-21T01:25:22.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128178 14.24 MB/sec execute 349 sec latency 9724.808 ms 2024-09-21T01:25:23.702 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '42f0107a', total seconds elapsed: 28.3 2024-09-21T01:25:23.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:23 smithi039 ceph-mon[30415]: pgmap v1077: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-09-21T01:25:23.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 128896 14.33 MB/sec execute 350 sec latency 96.520 ms 2024-09-21T01:25:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:23 smithi150 ceph-mon[34945]: pgmap v1077: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-09-21T01:25:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:23 smithi160 ceph-mon[34696]: pgmap v1077: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-09-21T01:25:24.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 129412 14.35 MB/sec execute 351 sec latency 126.192 ms 2024-09-21T01:25:25.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 129880 14.37 MB/sec execute 352 sec latency 143.687 ms 2024-09-21T01:25:26.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:25 smithi150 ceph-mon[34945]: pgmap v1078: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:25:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:25 smithi160 ceph-mon[34696]: pgmap v1078: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:25:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:25 smithi039 ceph-mon[30415]: pgmap v1078: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:25:26.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 130438 14.44 MB/sec execute 353 sec latency 112.895 ms 2024-09-21T01:25:27.907 INFO:tasks.workunit.client.0.smithi039.stdout: 10 130973 14.46 MB/sec execute 354 sec latency 125.084 ms 2024-09-21T01:25:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:27 smithi150 ceph-mon[34945]: pgmap v1079: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 89 op/s 2024-09-21T01:25:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:27 smithi160 ceph-mon[34696]: pgmap v1079: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 89 op/s 2024-09-21T01:25:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:27 smithi039 ceph-mon[30415]: pgmap v1079: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 89 op/s 2024-09-21T01:25:28.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 131762 14.55 MB/sec execute 355 sec latency 696.055 ms 2024-09-21T01:25:29.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 132398 14.62 MB/sec execute 356 sec latency 1696.108 ms 2024-09-21T01:25:30.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:29 smithi150 ceph-mon[34945]: pgmap v1080: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.0 MiB/s wr, 217 op/s 2024-09-21T01:25:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:29 smithi160 ceph-mon[34696]: pgmap v1080: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.0 MiB/s wr, 217 op/s 2024-09-21T01:25:30.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:29 smithi039 ceph-mon[30415]: pgmap v1080: 129 pgs: 129 active+clean; 1.6 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.0 MiB/s wr, 217 op/s 2024-09-21T01:25:30.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 132877 14.64 MB/sec execute 357 sec latency 2696.173 ms 2024-09-21T01:25:31.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 133361 14.67 MB/sec execute 358 sec latency 3696.235 ms 2024-09-21T01:25:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:31 smithi150 ceph-mon[34945]: pgmap v1081: 129 pgs: 129 active+clean; 1.6 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-09-21T01:25:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:31 smithi160 ceph-mon[34696]: pgmap v1081: 129 pgs: 129 active+clean; 1.6 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-09-21T01:25:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:31 smithi039 ceph-mon[30415]: pgmap v1081: 129 pgs: 129 active+clean; 1.6 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-09-21T01:25:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 133982 14.73 MB/sec execute 359 sec latency 4696.286 ms 2024-09-21T01:25:33.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 134701 14.80 MB/sec execute 360 sec latency 5696.346 ms 2024-09-21T01:25:34.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:33 smithi150 ceph-mon[34945]: pgmap v1082: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.9 MiB/s wr, 299 op/s 2024-09-21T01:25:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:33 smithi160 ceph-mon[34696]: pgmap v1082: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.9 MiB/s wr, 299 op/s 2024-09-21T01:25:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:33 smithi039 ceph-mon[30415]: pgmap v1082: 129 pgs: 129 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.9 MiB/s wr, 299 op/s 2024-09-21T01:25:34.909 INFO:tasks.workunit.client.0.smithi039.stdout: 10 135255 14.84 MB/sec execute 361 sec latency 6268.521 ms 2024-09-21T01:25:35.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 135769 14.88 MB/sec execute 362 sec latency 1301.820 ms 2024-09-21T01:25:36.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:35 smithi150 ceph-mon[34945]: pgmap v1083: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.2 MiB/s wr, 368 op/s 2024-09-21T01:25:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:35 smithi160 ceph-mon[34696]: pgmap v1083: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.2 MiB/s wr, 368 op/s 2024-09-21T01:25:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:35 smithi039 ceph-mon[30415]: pgmap v1083: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.2 MiB/s wr, 368 op/s 2024-09-21T01:25:36.908 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136232 14.90 MB/sec execute 363 sec latency 2143.239 ms 2024-09-21T01:25:37.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:37 smithi039 ceph-mon[30415]: pgmap v1084: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 417 op/s 2024-09-21T01:25:37.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:37 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:37.929 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136399 14.87 MB/sec execute 364 sec latency 454.364 ms 2024-09-21T01:25:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:37 smithi150 ceph-mon[34945]: pgmap v1084: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 417 op/s 2024-09-21T01:25:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:37 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:37 smithi160 ceph-mon[34696]: pgmap v1084: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 417 op/s 2024-09-21T01:25:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:37 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:38.941 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136401 14.83 MB/sec execute 365 sec latency 1475.345 ms 2024-09-21T01:25:39.947 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136401 14.79 MB/sec execute 366 sec latency 2481.806 ms 2024-09-21T01:25:39.969 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:39 smithi039 ceph-mon[30415]: pgmap v1085: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 473 op/s 2024-09-21T01:25:39.969 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:39.969 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:39.969 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:25:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:39 smithi150 ceph-mon[34945]: pgmap v1085: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 473 op/s 2024-09-21T01:25:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:25:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:39 smithi160 ceph-mon[34696]: pgmap v1085: 129 pgs: 129 active+clean; 1.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 473 op/s 2024-09-21T01:25:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:25:40.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136401 14.75 MB/sec execute 367 sec latency 3481.885 ms 2024-09-21T01:25:41.944 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:41 smithi039 ceph-mon[30415]: pgmap v1086: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 427 op/s 2024-09-21T01:25:41.947 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136409 14.71 MB/sec execute 368 sec latency 4481.944 ms 2024-09-21T01:25:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:41 smithi150 ceph-mon[34945]: pgmap v1086: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 427 op/s 2024-09-21T01:25:42.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:41 smithi160 ceph-mon[34696]: pgmap v1086: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 427 op/s 2024-09-21T01:25:42.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136409 14.67 MB/sec execute 369 sec latency 5481.999 ms 2024-09-21T01:25:43.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136463 14.64 MB/sec execute 370 sec latency 6430.892 ms 2024-09-21T01:25:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:43 smithi150 ceph-mon[34945]: pgmap v1087: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 12 MiB/s wr, 432 op/s 2024-09-21T01:25:44.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:43 smithi160 ceph-mon[34696]: pgmap v1087: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 12 MiB/s wr, 432 op/s 2024-09-21T01:25:44.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:43 smithi039 ceph-mon[30415]: pgmap v1087: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 12 MiB/s wr, 432 op/s 2024-09-21T01:25:44.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136463 14.60 MB/sec execute 371 sec latency 7430.942 ms 2024-09-21T01:25:45.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136463 14.56 MB/sec execute 372 sec latency 8430.987 ms 2024-09-21T01:25:46.055 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:45 smithi039 ceph-mon[30415]: pgmap v1088: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 12 MiB/s wr, 426 op/s 2024-09-21T01:25:46.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:45 smithi150 ceph-mon[34945]: pgmap v1088: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 12 MiB/s wr, 426 op/s 2024-09-21T01:25:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:45 smithi160 ceph-mon[34696]: pgmap v1088: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 12 MiB/s wr, 426 op/s 2024-09-21T01:25:46.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136463 14.52 MB/sec execute 373 sec latency 9431.032 ms 2024-09-21T01:25:47.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 136556 14.49 MB/sec execute 374 sec latency 10431.076 ms 2024-09-21T01:25:48.055 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:47 smithi039 ceph-mon[30415]: pgmap v1089: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 11 MiB/s wr, 373 op/s 2024-09-21T01:25:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:47 smithi150 ceph-mon[34945]: pgmap v1089: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 11 MiB/s wr, 373 op/s 2024-09-21T01:25:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:47 smithi160 ceph-mon[34696]: pgmap v1089: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 11 MiB/s wr, 373 op/s 2024-09-21T01:25:48.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 137117 14.54 MB/sec execute 375 sec latency 10565.596 ms 2024-09-21T01:25:49.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 137686 14.58 MB/sec execute 376 sec latency 113.270 ms 2024-09-21T01:25:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:49 smithi150 ceph-mon[34945]: pgmap v1090: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.4 MiB/s wr, 343 op/s 2024-09-21T01:25:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:49 smithi160 ceph-mon[34696]: pgmap v1090: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.4 MiB/s wr, 343 op/s 2024-09-21T01:25:50.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:49 smithi039 ceph-mon[30415]: pgmap v1090: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.4 MiB/s wr, 343 op/s 2024-09-21T01:25:50.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 138292 14.62 MB/sec execute 377 sec latency 102.503 ms 2024-09-21T01:25:51.948 INFO:tasks.workunit.client.0.smithi039.stdout: 10 138954 14.70 MB/sec execute 378 sec latency 130.703 ms 2024-09-21T01:25:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:51 smithi150 ceph-mon[34945]: pgmap v1091: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.4 MiB/s wr, 331 op/s 2024-09-21T01:25:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:51 smithi160 ceph-mon[34696]: pgmap v1091: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.4 MiB/s wr, 331 op/s 2024-09-21T01:25:52.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:51 smithi039 ceph-mon[30415]: pgmap v1091: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.4 MiB/s wr, 331 op/s 2024-09-21T01:25:52.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 139554 14.73 MB/sec execute 379 sec latency 188.924 ms 2024-09-21T01:25:53.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 140085 14.77 MB/sec execute 380 sec latency 207.090 ms 2024-09-21T01:25:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:53 smithi150 ceph-mon[34945]: pgmap v1092: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-09-21T01:25:54.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:53 smithi160 ceph-mon[34696]: pgmap v1092: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-09-21T01:25:54.155 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:53 smithi039 ceph-mon[30415]: pgmap v1092: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-09-21T01:25:54.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 140606 14.82 MB/sec execute 381 sec latency 173.543 ms 2024-09-21T01:25:55.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 141169 14.85 MB/sec execute 382 sec latency 111.642 ms 2024-09-21T01:25:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:55 smithi160 ceph-mon[34696]: pgmap v1093: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-09-21T01:25:56.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:55 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:25:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:55 smithi039 ceph-mon[30415]: pgmap v1093: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-09-21T01:25:56.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:55 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:25:56.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:55 smithi150 ceph-mon[34945]: pgmap v1093: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-09-21T01:25:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:55 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:25:56.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 141788 14.89 MB/sec execute 383 sec latency 141.552 ms 2024-09-21T01:25:57.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:57 smithi039 ceph-mon[30415]: pgmap v1094: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 287 op/s 2024-09-21T01:25:57.949 INFO:tasks.workunit.client.0.smithi039.stdout: 10 142350 14.95 MB/sec execute 384 sec latency 321.055 ms 2024-09-21T01:25:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:57 smithi160 ceph-mon[34696]: pgmap v1094: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 287 op/s 2024-09-21T01:25:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:57 smithi150 ceph-mon[34945]: pgmap v1094: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 287 op/s 2024-09-21T01:25:58.953 INFO:tasks.workunit.client.0.smithi039.stdout: 10 142499 14.94 MB/sec execute 385 sec latency 727.441 ms 2024-09-21T01:25:59.953 INFO:tasks.workunit.client.0.smithi039.stdout: 10 142603 14.91 MB/sec execute 386 sec latency 1633.345 ms 2024-09-21T01:26:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:25:59 smithi160 ceph-mon[34696]: pgmap v1095: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.6 MiB/s wr, 362 op/s 2024-09-21T01:26:00.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:25:59 smithi039 ceph-mon[30415]: pgmap v1095: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.6 MiB/s wr, 362 op/s 2024-09-21T01:26:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:25:59 smithi150 ceph-mon[34945]: pgmap v1095: 129 pgs: 129 active+clean; 1.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.6 MiB/s wr, 362 op/s 2024-09-21T01:26:00.953 INFO:tasks.workunit.client.0.smithi039.stdout: 10 142603 14.88 MB/sec execute 387 sec latency 2633.403 ms 2024-09-21T01:26:01.953 INFO:tasks.workunit.client.0.smithi039.stdout: 10 142606 14.84 MB/sec execute 388 sec latency 3633.458 ms 2024-09-21T01:26:02.106 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:01 smithi039 ceph-mon[30415]: pgmap v1096: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 7.6 MiB/s wr, 427 op/s 2024-09-21T01:26:02.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:01 smithi160 ceph-mon[34696]: pgmap v1096: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 7.6 MiB/s wr, 427 op/s 2024-09-21T01:26:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:01 smithi150 ceph-mon[34945]: pgmap v1096: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 7.6 MiB/s wr, 427 op/s 2024-09-21T01:26:02.954 INFO:tasks.workunit.client.0.smithi039.stdout: 10 142941 14.82 MB/sec execute 389 sec latency 3833.701 ms 2024-09-21T01:26:03.895 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:03 smithi039 ceph-mon[30415]: pgmap v1097: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.0 MiB/s wr, 414 op/s 2024-09-21T01:26:03.966 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143119 14.80 MB/sec execute 390 sec latency 785.888 ms 2024-09-21T01:26:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:03 smithi150 ceph-mon[34945]: pgmap v1097: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.0 MiB/s wr, 414 op/s 2024-09-21T01:26:04.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:03 smithi160 ceph-mon[34696]: pgmap v1097: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 8.0 MiB/s wr, 414 op/s 2024-09-21T01:26:04.967 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143204 14.78 MB/sec execute 391 sec latency 1639.203 ms 2024-09-21T01:26:05.117 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.8 2024-09-21T01:26:05.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:26:05.967 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143278 14.76 MB/sec execute 392 sec latency 2118.025 ms 2024-09-21T01:26:06.013 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:26:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:05 smithi039 ceph-mon[30415]: pgmap v1098: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.7 MiB/s wr, 450 op/s 2024-09-21T01:26:06.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:05 smithi039 ceph-mon[30415]: osdmap e123: 12 total, 12 up, 12 in 2024-09-21T01:26:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:05 smithi150 ceph-mon[34945]: pgmap v1098: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.7 MiB/s wr, 450 op/s 2024-09-21T01:26:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:05 smithi150 ceph-mon[34945]: osdmap e123: 12 total, 12 up, 12 in 2024-09-21T01:26:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:05 smithi160 ceph-mon[34696]: pgmap v1098: 129 pgs: 129 active+clean; 1.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.7 MiB/s wr, 450 op/s 2024-09-21T01:26:06.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:05 smithi160 ceph-mon[34696]: osdmap e123: 12 total, 12 up, 12 in 2024-09-21T01:26:06.967 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143347 14.73 MB/sec execute 393 sec latency 2980.037 ms 2024-09-21T01:26:07.967 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143347 14.70 MB/sec execute 394 sec latency 3980.090 ms 2024-09-21T01:26:08.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:07 smithi039 ceph-mon[30415]: pgmap v1100: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.2 MiB/s wr, 420 op/s 2024-09-21T01:26:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:07 smithi150 ceph-mon[34945]: pgmap v1100: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.2 MiB/s wr, 420 op/s 2024-09-21T01:26:08.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:07 smithi160 ceph-mon[34696]: pgmap v1100: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.2 MiB/s wr, 420 op/s 2024-09-21T01:26:08.967 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143347 14.66 MB/sec execute 395 sec latency 4980.164 ms 2024-09-21T01:26:09.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143458 14.64 MB/sec execute 396 sec latency 5374.778 ms 2024-09-21T01:26:10.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:09 smithi039 ceph-mon[30415]: pgmap v1101: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 10 MiB/s wr, 398 op/s 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 -- 172.21.15.39:0/2926554415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 msgr2=0x7f41ac107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 --2- 172.21.15.39:0/2926554415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 0x7f41ac107620 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f419801a910 tx=0x7f419803b430 comp rx=0 tx=0).stop 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 -- 172.21.15.39:0/2926554415 shutdown_connections 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 --2- 172.21.15.39:0/2926554415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 --2- 172.21.15.39:0/2926554415 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 0x7f41ac107620 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 --2- 172.21.15.39:0/2926554415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac1027a0 0x7f41ac102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.777+0000 7f41b0a00640 1 -- 172.21.15.39:0/2926554415 >> 172.21.15.39:0/2926554415 conn(0x7f41ac079730 msgr2=0x7f41ac079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.778+0000 7f41b0a00640 1 -- 172.21.15.39:0/2926554415 shutdown_connections 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.778+0000 7f41b0a00640 1 -- 172.21.15.39:0/2926554415 wait complete. 2024-09-21T01:26:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 Processor -- start 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 -- start start 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac1027a0 0x7f41ac116d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 0x7f41ac113010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10b4e0 con 0x7f41ac107be0 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.780+0000 7f41b0a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10b360 con 0x7f41ac1027a0 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.781+0000 7f41b0a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10b660 con 0x7f41ac1030f0 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.781+0000 7f41aa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.782+0000 7f41a9e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac1027a0 0x7f41ac116d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.782+0000 7f41aa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41572/0 (socket says 172.21.15.39:41572) 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.782+0000 7f41aa800640 1 -- 172.21.15.39:0/1336871727 learned_addr learned my addr 172.21.15.39:0/1336871727 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.787+0000 7f41aa800640 1 -- 172.21.15.39:0/1336871727 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 msgr2=0x7f41ac113010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.787+0000 7f41aa800640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 0x7f41ac113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.787+0000 7f41aa800640 1 -- 172.21.15.39:0/1336871727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac1027a0 msgr2=0x7f41ac116d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.787+0000 7f41aa800640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac1027a0 0x7f41ac116d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.787+0000 7f41aa800640 1 -- 172.21.15.39:0/1336871727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac113e20 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.787+0000 7f41aa800640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac113600 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f41a400b970 tx=0x7f41a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.789+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4019070 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.789+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac11f9b0 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.789+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac11ff40 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.789+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41a4014440 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.789+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a40075b0 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.791+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f418c0052c0 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.804+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f41a401c070 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.804+0000 7f41a2a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41900b67d0 0x7f41900b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.809+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9599+0+0 (secure 0 0 0) 0x7f41a40e72a0 con 0x7f41ac107be0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.809+0000 7f41a9e00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41900b67d0 0x7f41900b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:10.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.810+0000 7f41a9e00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41900b67d0 0x7f41900b8ce0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f419c005fd0 tx=0x7f419c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:09.816+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f41a40ad440 con 0x7f41ac107be0 2024-09-21T01:26:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.278+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f418c0055b0 con 0x7f41ac107be0 2024-09-21T01:26:10.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.278+0000 7f41a2a00640 1 -- 172.21.15.39:0/1336871727 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+11917 (secure 0 0 0) 0x7f41a40b1820 con 0x7f41ac107be0 2024-09-21T01:26:10.282 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:26:10.283 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":61,"btime":"2024-09-21T01:25:54:810434+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":61,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:25:54.408578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:26:10.283 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 61 2024-09-21T01:26:10.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41900b67d0 msgr2=0x7f41900b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:10.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41900b67d0 0x7f41900b8ce0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f419c005fd0 tx=0x7f419c005850 comp rx=0 tx=0).stop 2024-09-21T01:26:10.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 msgr2=0x7f41ac113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:10.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac113600 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f41a400b970 tx=0x7f41a400be30 comp rx=0 tx=0).stop 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 shutdown_connections 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107be0 0x7f41ac113600 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f41ac1030f0 0x7f41ac113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41900b67d0 0x7f41900b8ce0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 --2- 172.21.15.39:0/1336871727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac1027a0 0x7f41ac116d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 >> 172.21.15.39:0/1336871727 conn(0x7f41ac079730 msgr2=0x7f41ac1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 shutdown_connections 2024-09-21T01:26:10.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:10.285+0000 7f41b0a00640 1 -- 172.21.15.39:0/1336871727 wait complete. 2024-09-21T01:26:10.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:09 smithi150 ceph-mon[34945]: pgmap v1101: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 10 MiB/s wr, 398 op/s 2024-09-21T01:26:10.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:09 smithi160 ceph-mon[34696]: pgmap v1101: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 10 MiB/s wr, 398 op/s 2024-09-21T01:26:10.977 INFO:tasks.workunit.client.0.smithi039.stdout: 10 143846 14.65 MB/sec execute 397 sec latency 616.990 ms 2024-09-21T01:26:11.251 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:10 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1336871727' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:26:11.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:10 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1336871727' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:26:11.319 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 149.8 --await-for 45.0' 2024-09-21T01:26:11.319 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 149.8 --await-for 45.0 2024-09-21T01:26:11.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:10 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1336871727' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:26:11.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 144287 14.66 MB/sec execute 398 sec latency 121.175 ms 2024-09-21T01:26:12.020 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:26:12.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:11 smithi039 ceph-mon[30415]: pgmap v1102: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 MiB/s wr, 329 op/s 2024-09-21T01:26:12.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:11 smithi150 ceph-mon[34945]: pgmap v1102: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 MiB/s wr, 329 op/s 2024-09-21T01:26:12.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:11 smithi160 ceph-mon[34696]: pgmap v1102: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 MiB/s wr, 329 op/s 2024-09-21T01:26:12.977 INFO:tasks.workunit.client.0.smithi039.stdout: 10 144697 14.67 MB/sec execute 399 sec latency 230.249 ms 2024-09-21T01:26:13.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145227 14.72 MB/sec execute 400 sec latency 229.212 ms 2024-09-21T01:26:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:13 smithi039 ceph-mon[30415]: pgmap v1103: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 308 op/s 2024-09-21T01:26:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:13 smithi150 ceph-mon[34945]: pgmap v1103: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 308 op/s 2024-09-21T01:26:14.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:13 smithi160 ceph-mon[34696]: pgmap v1103: 129 pgs: 129 active+clean; 1.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 308 op/s 2024-09-21T01:26:14.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145726 14.77 MB/sec execute 401 sec latency 261.634 ms 2024-09-21T01:26:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.706+0000 7fab1e200640 1 -- 172.21.15.39:0/3742191153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab1806af90 msgr2=0x7fab1810cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.706+0000 7fab1e200640 1 --2- 172.21.15.39:0/3742191153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab1806af90 0x7fab1810cb00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fab0800dc90 tx=0x7fab0802e370 comp rx=0 tx=0).stop 2024-09-21T01:26:15.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.710+0000 7fab1e200640 1 -- 172.21.15.39:0/3742191153 shutdown_connections 2024-09-21T01:26:15.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.710+0000 7fab1e200640 1 --2- 172.21.15.39:0/3742191153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab18114eb0 0x7fab181172d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.710+0000 7fab1e200640 1 --2- 172.21.15.39:0/3742191153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 0x7fab1810d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.710+0000 7fab1e200640 1 --2- 172.21.15.39:0/3742191153 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab1806af90 0x7fab1810cb00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.710+0000 7fab1e200640 1 -- 172.21.15.39:0/3742191153 >> 172.21.15.39:0/3742191153 conn(0x7fab1806ee90 msgr2=0x7fab1806f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:15.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.712+0000 7fab1e200640 1 -- 172.21.15.39:0/3742191153 shutdown_connections 2024-09-21T01:26:15.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.712+0000 7fab1e200640 1 -- 172.21.15.39:0/3742191153 wait complete. 2024-09-21T01:26:15.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.714+0000 7fab1e200640 1 Processor -- start 2024-09-21T01:26:15.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.715+0000 7fab1e200640 1 -- start start 2024-09-21T01:26:15.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab1806af90 0x7fab181a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1e200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 0x7fab181a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1e200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab18114eb0 0x7fab181a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fab18119f40 con 0x7fab1806af90 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1e200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fab18119dc0 con 0x7fab1810afc0 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1e200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fab1811a0c0 con 0x7fab18114eb0 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab1806af90 0x7fab181a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.716+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab1806af90 0x7fab181a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57310/0 (socket says 172.21.15.39:57310) 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.717+0000 7fab1ce00640 1 -- 172.21.15.39:0/3363623704 learned_addr learned my addr 172.21.15.39:0/3363623704 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.717+0000 7fab0fe00640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 0x7fab181a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.717+0000 7fab1d800640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab18114eb0 0x7fab181a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.718+0000 7fab1d800640 1 -- 172.21.15.39:0/3363623704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 msgr2=0x7fab181a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.718+0000 7fab1d800640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 0x7fab181a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.718+0000 7fab1d800640 1 -- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab1806af90 msgr2=0x7fab181a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.718+0000 7fab1d800640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab1806af90 0x7fab181a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.718+0000 7fab1d800640 1 -- 172.21.15.39:0/3363623704 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab181a68c0 con 0x7fab18114eb0 2024-09-21T01:26:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.719+0000 7fab0fe00640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 0x7fab181a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:15.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.720+0000 7fab1d800640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab18114eb0 0x7fab181a6190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fab10009fd0 tx=0x7fab1000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:15.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.720+0000 7fab16000640 1 -- 172.21.15.39:0/3363623704 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000ce20 con 0x7fab18114eb0 2024-09-21T01:26:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.721+0000 7fab16000640 1 -- 172.21.15.39:0/3363623704 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab10005020 con 0x7fab18114eb0 2024-09-21T01:26:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.721+0000 7fab16000640 1 -- 172.21.15.39:0/3363623704 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10010690 con 0x7fab18114eb0 2024-09-21T01:26:15.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.721+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab181a6bb0 con 0x7fab18114eb0 2024-09-21T01:26:15.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.721+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab181ad310 con 0x7fab18114eb0 2024-09-21T01:26:15.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.732+0000 7fab2649c740 1 Processor -- start 2024-09-21T01:26:15.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.733+0000 7fab16000640 1 -- 172.21.15.39:0/3363623704 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fab10010830 con 0x7fab18114eb0 2024-09-21T01:26:15.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.734+0000 7fab2649c740 1 -- start start 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab16000640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faaf80b6770 0x7faaf80b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab2649c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 0x557497c43510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab2649c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c43a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab2649c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497c43f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab2649c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557497bb8ca0 con 0x55749789b840 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab2649c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557497ba80d0 con 0x55749787ab60 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.736+0000 7fab2649c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557497babee0 con 0x557497872980 2024-09-21T01:26:15.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.737+0000 7fab16000640 1 -- 172.21.15.39:0/3363623704 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7fab100e81e0 con 0x7fab18114eb0 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.737+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 0x557497c43510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.737+0000 7fab0fe00640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faaf80b6770 0x7faaf80b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.737+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 0x557497c43510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57326/0 (socket says 172.21.15.39:57326) 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.737+0000 7fab1ce00640 1 -- 172.21.15.39:0/398177930 learned_addr learned my addr 172.21.15.39:0/398177930 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.737+0000 7fab0fe00640 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c43a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.738+0000 7fab1d800640 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497c43f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.738+0000 7fab1ce00640 1 -- 172.21.15.39:0/398177930 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 msgr2=0x557497c43a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.738+0000 7fab1ce00640 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c43a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.738+0000 7fab1ce00640 1 -- 172.21.15.39:0/398177930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 msgr2=0x557497c43f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.738+0000 7fab1ce00640 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497c43f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.738+0000 7fab1ce00640 1 -- 172.21.15.39:0/398177930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557497b5a400 con 0x55749789b840 2024-09-21T01:26:15.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.739+0000 7fab1ce00640 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 0x557497c43510 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fab08005d50 tx=0x7fab08005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d742d40b6e3493 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:15.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.739+0000 7fab0e000640 1 -- 172.21.15.39:0/398177930 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080046c0 con 0x55749789b840 2024-09-21T01:26:15.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.739+0000 7fab0e000640 1 -- 172.21.15.39:0/398177930 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab08043070 con 0x55749789b840 2024-09-21T01:26:15.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.739+0000 7fab0e000640 1 -- 172.21.15.39:0/398177930 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080052e0 con 0x55749789b840 2024-09-21T01:26:15.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 -- 172.21.15.39:0/398177930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 msgr2=0x557497c43510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:15.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 0x557497c43510 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fab08005d50 tx=0x7fab08005d80 comp rx=0 tx=0).stop 2024-09-21T01:26:15.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 -- 172.21.15.39:0/398177930 shutdown_connections 2024-09-21T01:26:15.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497c43f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c43a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 --2- 172.21.15.39:0/398177930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749789b840 0x557497c43510 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.741+0000 7fab2649c740 1 -- 172.21.15.39:0/398177930 >> 172.21.15.39:0/398177930 conn(0x55749788f990 msgr2=0x557497c41400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:15.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.742+0000 7fab2649c740 1 -- 172.21.15.39:0/398177930 shutdown_connections 2024-09-21T01:26:15.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.742+0000 7fab2649c740 1 -- 172.21.15.39:0/398177930 wait complete. 2024-09-21T01:26:15.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.743+0000 7fab2649c740 1 -- start start 2024-09-21T01:26:15.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.743+0000 7fab2649c740 1 Processor -- start 2024-09-21T01:26:15.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.744+0000 7fab0fe00640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faaf80b6770 0x7faaf80b8c80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x557497c42190 tx=0x7fab00009d30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.745+0000 7fab2649c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c4ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.745+0000 7fab2649c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749787ab60 0x557497c4b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab2649c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c637c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab2649c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557497a097a0 con 0x55749787ab60 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab2649c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557497b4da10 con 0x55749785a1c0 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab2649c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x557497c5a4a0 con 0x557497872980 2024-09-21T01:26:15.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab2649c740 10 client.0 _ll_register_callbacks cb 0x557497b5dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:26:15.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c4ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.746+0000 7fab0fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749787ab60 0x557497c4b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.747+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c4ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:37606/0 (socket says 172.21.15.39:37606) 2024-09-21T01:26:15.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.747+0000 7fab1ce00640 1 -- 172.21.15.39:0/497848698 learned_addr learned my addr 172.21.15.39:0/497848698 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:15.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab1d800640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c637c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.747+0000 7fab1ce00640 1 -- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 msgr2=0x557497c637c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab1ce00640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c637c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab1ce00640 1 -- 172.21.15.39:0/497848698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749787ab60 msgr2=0x557497c4b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab1ce00640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749787ab60 0x557497c4b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab1ce00640 1 -- 172.21.15.39:0/497848698 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557497915b90 con 0x557497872980 2024-09-21T01:26:15.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab0fe00640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749787ab60 0x557497c4b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:26:15.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.748+0000 7fab1d800640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c637c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:26:15.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.749+0000 7fab1ce00640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c4ac70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab08000c00 tx=0x7fab08004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7363f397008c6d41 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:15.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.749+0000 7fab06a00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:26:15.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.754+0000 7fab06a00640 1 -- 172.21.15.39:0/497848698 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08005060 con 0x557497872980 2024-09-21T01:26:15.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.754+0000 7fab06a00640 1 -- 172.21.15.39:0/497848698 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7fab08005200 con 0x557497872980 2024-09-21T01:26:15.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.755+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab08043070 con 0x557497872980 2024-09-21T01:26:15.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.755+0000 7fab06a00640 1 -- 172.21.15.39:0/497848698 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fab08005400 con 0x557497872980 2024-09-21T01:26:15.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.755+0000 7fab2649c740 10 client.26434 fetch_fsmap learned FSMap version 61 2024-09-21T01:26:15.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.755+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x557497c662b0 con 0x557497872980 2024-09-21T01:26:15.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.756+0000 7fab06a00640 1 -- 172.21.15.39:0/497848698 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab08043070 con 0x557497872980 2024-09-21T01:26:15.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab06a00640 1 -- 172.21.15.39:0/497848698 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 61) ==== 8370+0+0 (secure 0 0 0) 0x7fab0803e410 con 0x557497872980 2024-09-21T01:26:15.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab2649c740 10 client.26434 fetch_fsmap finished waiting for FSMap version 61 2024-09-21T01:26:15.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab2649c740 10 client.26434 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:26:15.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab2649c740 20 client.26434 populate_metadata read hostname 'smithi039' 2024-09-21T01:26:15.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab2649c740 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749782f570 0x557497c69610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab2649c740 4 client.26434 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:26:15.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557497bbabf0 con 0x55749782f570 2024-09-21T01:26:15.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.757+0000 7fab0fe00640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749782f570 0x557497c69610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.758+0000 7fab0fe00640 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749782f570 0x557497c69610 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6713fa2b0509d0b server_cookie=9409bb5a45b1db0e in_seq=0 out_seq=0 2024-09-21T01:26:15.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.758+0000 7fab06a00640 10 client.26434 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:15.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.759+0000 7fab06a00640 1 -- 172.21.15.39:0/497848698 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557497bbabf0 con 0x55749782f570 2024-09-21T01:26:15.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.759+0000 7fab06a00640 10 client.26434 handle_command_reply: tid=0 2024-09-21T01:26:15.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.759+0000 7fab2649c740 1 client.26434 shutdown 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 msgr2=0x557497c4ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c4ac70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab08000c00 tx=0x7fab08004e40 comp rx=0 tx=0).stop 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 shutdown_connections 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55749782f570 0x557497c69610 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c637c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749787ab60 0x557497c4b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 --2- 172.21.15.39:0/497848698 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497c4ac70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 >> 172.21.15.39:0/497848698 conn(0x55749788f990 msgr2=0x557497c466f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab06a00640 0 client.26434 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:15.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 shutdown_connections 2024-09-21T01:26:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 1 -- 172.21.15.39:0/497848698 wait complete. 2024-09-21T01:26:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.760+0000 7fab2649c740 20 client.26434 trim_cache size 0 max 16384 2024-09-21T01:26:15.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.786+0000 7fab2649c740 1 Processor -- start 2024-09-21T01:26:15.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.786+0000 7fab2649c740 1 -- start start 2024-09-21T01:26:15.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.787+0000 7fab2649c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c46bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.787+0000 7fab2649c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557497872980 0x557497c484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.787+0000 7fab2649c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749787ab60 0x557497c44c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.787+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c46bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.787+0000 7fab1ce00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c46bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50908/0 (socket says 172.21.15.39:50908) 2024-09-21T01:26:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.787+0000 7fab1ce00640 1 -- 172.21.15.39:0/72581959 learned_addr learned my addr 172.21.15.39:0/72581959 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab0fe00640 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557497872980 0x557497c484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b9cd0 con 0x557497872980 2024-09-21T01:26:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b7540 con 0x55749785a1c0 2024-09-21T01:26:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b53b0 con 0x55749787ab60 2024-09-21T01:26:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab1d800640 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749787ab60 0x557497c44c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab0fe00640 1 -- 172.21.15.39:0/72581959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749787ab60 msgr2=0x557497c44c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab0fe00640 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749787ab60 0x557497c44c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab0fe00640 1 -- 172.21.15.39:0/72581959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 msgr2=0x557497c46bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab0fe00640 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c46bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.788+0000 7fab0fe00640 1 -- 172.21.15.39:0/72581959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557497c421c0 con 0x557497872980 2024-09-21T01:26:15.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.789+0000 7fab1d800640 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749787ab60 0x557497c44c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:15.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.789+0000 7fab0fe00640 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557497872980 0x557497c484f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x557497b4db60 tx=0x7fab00005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a983117911e9ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.790+0000 7fab0d600640 1 -- 172.21.15.39:0/72581959 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0000fd80 con 0x557497872980 2024-09-21T01:26:15.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.790+0000 7fab0d600640 1 -- 172.21.15.39:0/72581959 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab0002d070 con 0x557497872980 2024-09-21T01:26:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.790+0000 7fab0d600640 1 -- 172.21.15.39:0/72581959 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00003de0 con 0x557497872980 2024-09-21T01:26:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.791+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557497872980 msgr2=0x557497c484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.791+0000 7fab2649c740 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557497872980 0x557497c484f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x557497b4db60 tx=0x7fab00005f90 comp rx=0 tx=0).stop 2024-09-21T01:26:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 shutdown_connections 2024-09-21T01:26:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55749787ab60 0x557497c44c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557497872980 0x557497c484f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 --2- 172.21.15.39:0/72581959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749785a1c0 0x557497c46bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 >> 172.21.15.39:0/72581959 conn(0x55749788f990 msgr2=0x557497c44540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 shutdown_connections 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.792+0000 7fab2649c740 1 -- 172.21.15.39:0/72581959 wait complete. 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.793+0000 7fab2649c740 1 -- start start 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.793+0000 7fab2649c740 1 Processor -- start 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749785a1c0 0x557497cc7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497cc84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497cca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b5a00 con 0x55749785a1c0 2024-09-21T01:26:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b6e70 con 0x55749787ab60 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b60d0 con 0x557497872980 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab2649c740 10 client.0 _ll_register_callbacks cb 0x7fab0800db90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab1d800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497cca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab0fe00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497cc84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab1d800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497cca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50912/0 (socket says 172.21.15.39:50912) 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab1d800640 1 -- 172.21.15.39:0/2109064657 learned_addr learned my addr 172.21.15.39:0/2109064657 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.794+0000 7fab1ce00640 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749785a1c0 0x557497cc7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.795+0000 7fab1d800640 1 -- 172.21.15.39:0/2109064657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 msgr2=0x557497cc84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.795+0000 7fab1d800640 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497cc84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.795+0000 7fab1d800640 1 -- 172.21.15.39:0/2109064657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749785a1c0 msgr2=0x557497cc7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.795+0000 7fab1d800640 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749785a1c0 0x557497cc7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.795+0000 7fab1d800640 1 -- 172.21.15.39:0/2109064657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557497cca680 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.796+0000 7fab1d800640 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497cca010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab100adb80 tx=0x7fab100aa2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1859aeb0ee7bbcd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.796+0000 7fab06a00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.797+0000 7fab06a00640 1 -- 172.21.15.39:0/2109064657 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100e5970 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.797+0000 7fab06a00640 1 -- 172.21.15.39:0/2109064657 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7fab100b21e0 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.797+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab0002d070 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.797+0000 7fab06a00640 1 -- 172.21.15.39:0/2109064657 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fab100ae2f0 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.797+0000 7fab2649c740 10 client.26462 fetch_fsmap learned FSMap version 61 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.797+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x557497cd4cf0 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab06a00640 1 -- 172.21.15.39:0/2109064657 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab0002d070 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab06a00640 1 -- 172.21.15.39:0/2109064657 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 61) ==== 8370+0+0 (secure 0 0 0) 0x7fab100a9040 con 0x55749787ab60 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab2649c740 10 client.26462 fetch_fsmap finished waiting for FSMap version 61 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab2649c740 10 client.26462 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:26:15.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab2649c740 20 client.26462 populate_metadata read hostname 'smithi039' 2024-09-21T01:26:15.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab2649c740 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557497cd5490 0x557497cd78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:15.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab2649c740 4 client.26462 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.8, "await_for": 45.0}] 2024-09-21T01:26:15.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.8, "await_for": 45.0}) -- 0x557497bb66a0 con 0x557497cd5490 2024-09-21T01:26:15.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.798+0000 7fab1ce00640 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557497cd5490 0x557497cd78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:15.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.800+0000 7fab1ce00640 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557497cd5490 0x557497cd78b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26f8d1b2973ba05f server_cookie=a630b501f6b7103d in_seq=0 out_seq=0 2024-09-21T01:26:15.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:15.800+0000 7fab06a00640 10 client.26462 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:15.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145936 14.76 MB/sec execute 402 sec latency 231.588 ms 2024-09-21T01:26:15.992 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:15 smithi039 ceph-mon[30415]: pgmap v1104: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.4 MiB/s wr, 320 op/s 2024-09-21T01:26:15.992 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:15 smithi039 ceph-mon[30415]: osdmap e124: 12 total, 12 up, 12 in 2024-09-21T01:26:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:15 smithi150 ceph-mon[34945]: pgmap v1104: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.4 MiB/s wr, 320 op/s 2024-09-21T01:26:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:15 smithi150 ceph-mon[34945]: osdmap e124: 12 total, 12 up, 12 in 2024-09-21T01:26:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:15 smithi160 ceph-mon[34696]: pgmap v1104: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.4 MiB/s wr, 320 op/s 2024-09-21T01:26:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:15 smithi160 ceph-mon[34696]: osdmap e124: 12 total, 12 up, 12 in 2024-09-21T01:26:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.289+0000 7fab06a00640 1 -- 172.21.15.39:0/2109064657 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557497bb66a0 con 0x557497cd5490 2024-09-21T01:26:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.289+0000 7fab06a00640 10 client.26462 handle_command_reply: tid=0 2024-09-21T01:26:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.289+0000 7fab2649c740 1 client.26462 shutdown 2024-09-21T01:26:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 msgr2=0x557497cca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497cca010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab100adb80 tx=0x7fab100aa2c0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 shutdown_connections 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x557497cd5490 0x557497cd78b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55749787ab60 0x557497cca010 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x557497872980 0x557497cc84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 --2- 172.21.15.39:0/2109064657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55749785a1c0 0x557497cc7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 >> 172.21.15.39:0/2109064657 conn(0x55749788f990 msgr2=0x557497c47480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:16.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab06a00640 0 client.26462 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.290+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 shutdown_connections 2024-09-21T01:26:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.291+0000 7fab2649c740 1 -- 172.21.15.39:0/2109064657 wait complete. 2024-09-21T01:26:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.291+0000 7fab2649c740 20 client.26462 trim_cache size 0 max 16384 2024-09-21T01:26:16.293 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:26:16.293 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 61, 2024-09-21T01:26:16.293 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 86, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "a2eb49a5": { 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 86, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 149.8, 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:26:16.294 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:16.295 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:16.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:16.296 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:26:16.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faaf80b6770 msgr2=0x7faaf80b8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:16.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faaf80b6770 0x7faaf80b8c80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x557497c42190 tx=0x7fab00009d30 comp rx=0 tx=0).stop 2024-09-21T01:26:16.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab18114eb0 msgr2=0x7fab181a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:16.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab18114eb0 0x7fab181a6190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fab10009fd0 tx=0x7fab1000ef10 comp rx=0 tx=0).stop 2024-09-21T01:26:16.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 shutdown_connections 2024-09-21T01:26:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab18114eb0 0x7fab181a6190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7faaf80b6770 0x7faaf80b8c80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab1810afc0 0x7fab181a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 --2- 172.21.15.39:0/3363623704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab1806af90 0x7fab181a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.292+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 >> 172.21.15.39:0/3363623704 conn(0x7fab1806ee90 msgr2=0x7fab18071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:16.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.293+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 shutdown_connections 2024-09-21T01:26:16.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:16.293+0000 7fab1e200640 1 -- 172.21.15.39:0/3363623704 wait complete. 2024-09-21T01:26:16.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.73 MB/sec execute 403 sec latency 1169.576 ms 2024-09-21T01:26:17.253 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a2eb49a5', quiesce took 12.1 seconds. Will release after: 17.700000000000003 2024-09-21T01:26:17.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.69 MB/sec execute 404 sec latency 2169.632 ms 2024-09-21T01:26:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:17 smithi150 ceph-mon[34945]: pgmap v1106: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 317 op/s 2024-09-21T01:26:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:17 smithi160 ceph-mon[34696]: pgmap v1106: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 317 op/s 2024-09-21T01:26:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:17 smithi039 ceph-mon[30415]: pgmap v1106: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 317 op/s 2024-09-21T01:26:18.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.65 MB/sec execute 405 sec latency 3169.711 ms 2024-09-21T01:26:19.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.62 MB/sec execute 406 sec latency 4169.864 ms 2024-09-21T01:26:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:19 smithi150 ceph-mon[34945]: pgmap v1107: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 9.2 MiB/s wr, 334 op/s 2024-09-21T01:26:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:19 smithi160 ceph-mon[34696]: pgmap v1107: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 9.2 MiB/s wr, 334 op/s 2024-09-21T01:26:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:19 smithi039 ceph-mon[30415]: pgmap v1107: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 9.2 MiB/s wr, 334 op/s 2024-09-21T01:26:20.978 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.58 MB/sec execute 407 sec latency 5169.956 ms 2024-09-21T01:26:21.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.55 MB/sec execute 408 sec latency 6170.126 ms 2024-09-21T01:26:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:21 smithi150 ceph-mon[34945]: pgmap v1108: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.4 MiB/s wr, 328 op/s 2024-09-21T01:26:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:21 smithi160 ceph-mon[34696]: pgmap v1108: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.4 MiB/s wr, 328 op/s 2024-09-21T01:26:22.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:21 smithi039 ceph-mon[30415]: pgmap v1108: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.4 MiB/s wr, 328 op/s 2024-09-21T01:26:22.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.51 MB/sec execute 409 sec latency 7170.280 ms 2024-09-21T01:26:23.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.48 MB/sec execute 410 sec latency 8170.377 ms 2024-09-21T01:26:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:23 smithi150 ceph-mon[34945]: pgmap v1109: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 292 op/s 2024-09-21T01:26:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:23 smithi160 ceph-mon[34696]: pgmap v1109: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 292 op/s 2024-09-21T01:26:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:23 smithi039 ceph-mon[30415]: pgmap v1109: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 292 op/s 2024-09-21T01:26:24.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.44 MB/sec execute 411 sec latency 9170.464 ms 2024-09-21T01:26:25.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.41 MB/sec execute 412 sec latency 10170.547 ms 2024-09-21T01:26:26.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:25 smithi039 ceph-mon[30415]: pgmap v1110: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.1 MiB/s wr, 208 op/s 2024-09-21T01:26:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:25 smithi150 ceph-mon[34945]: pgmap v1110: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.1 MiB/s wr, 208 op/s 2024-09-21T01:26:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:25 smithi160 ceph-mon[34696]: pgmap v1110: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.1 MiB/s wr, 208 op/s 2024-09-21T01:26:26.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.37 MB/sec execute 413 sec latency 11170.652 ms 2024-09-21T01:26:27.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.34 MB/sec execute 414 sec latency 12170.774 ms 2024-09-21T01:26:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:27 smithi150 ceph-mon[34945]: pgmap v1111: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 175 op/s 2024-09-21T01:26:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:27 smithi160 ceph-mon[34696]: pgmap v1111: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 175 op/s 2024-09-21T01:26:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:27 smithi039 ceph-mon[30415]: pgmap v1111: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 175 op/s 2024-09-21T01:26:28.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.30 MB/sec execute 415 sec latency 13170.929 ms 2024-09-21T01:26:29.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.27 MB/sec execute 416 sec latency 14171.096 ms 2024-09-21T01:26:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:29 smithi150 ceph-mon[34945]: pgmap v1112: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-09-21T01:26:30.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:29 smithi160 ceph-mon[34696]: pgmap v1112: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-09-21T01:26:30.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:29 smithi039 ceph-mon[30415]: pgmap v1112: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-09-21T01:26:30.979 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.23 MB/sec execute 417 sec latency 15171.199 ms 2024-09-21T01:26:31.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.20 MB/sec execute 418 sec latency 16171.346 ms 2024-09-21T01:26:32.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:31 smithi150 ceph-mon[34945]: pgmap v1113: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 781 KiB/s wr, 44 op/s 2024-09-21T01:26:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:31 smithi160 ceph-mon[34696]: pgmap v1113: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 781 KiB/s wr, 44 op/s 2024-09-21T01:26:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:31 smithi039 ceph-mon[30415]: pgmap v1113: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 781 KiB/s wr, 44 op/s 2024-09-21T01:26:32.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.16 MB/sec execute 419 sec latency 17171.423 ms 2024-09-21T01:26:33.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.13 MB/sec execute 420 sec latency 18171.569 ms 2024-09-21T01:26:34.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:33 smithi150 ceph-mon[34945]: pgmap v1114: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 182 KiB/s wr, 14 op/s 2024-09-21T01:26:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:33 smithi160 ceph-mon[34696]: pgmap v1114: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 182 KiB/s wr, 14 op/s 2024-09-21T01:26:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:33 smithi039 ceph-mon[30415]: pgmap v1114: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 182 KiB/s wr, 14 op/s 2024-09-21T01:26:34.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:26:34.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.10 MB/sec execute 421 sec latency 19171.660 ms 2024-09-21T01:26:35.352 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:26:35.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.970+0000 7fcf63e00640 1 -- 172.21.15.39:0/707716815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 msgr2=0x7fcf64075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:35.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.970+0000 7fcf63e00640 1 --2- 172.21.15.39:0/707716815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf64075bc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009900 tx=0x7fcf5402a410 comp rx=0 tx=0).stop 2024-09-21T01:26:35.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 -- 172.21.15.39:0/707716815 shutdown_connections 2024-09-21T01:26:35.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 --2- 172.21.15.39:0/707716815 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcf6410ff10 0x7fcf64112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:35.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 --2- 172.21.15.39:0/707716815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf64076100 0x7fcf6410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:35.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 --2- 172.21.15.39:0/707716815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf64075bc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:35.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 -- 172.21.15.39:0/707716815 >> 172.21.15.39:0/707716815 conn(0x7fcf640ff440 msgr2=0x7fcf64101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:35.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 -- 172.21.15.39:0/707716815 shutdown_connections 2024-09-21T01:26:35.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.971+0000 7fcf63e00640 1 -- 172.21.15.39:0/707716815 wait complete. 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 Processor -- start 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 -- start start 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf6419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcf64076100 0x7fcf6419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6410ff10 0x7fcf641a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64115ac0 con 0x7fcf640757b0 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64115940 con 0x7fcf6410ff10 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.972+0000 7fcf63e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64115c40 con 0x7fcf64076100 2024-09-21T01:26:35.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf62a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf6419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:35.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf63400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6410ff10 0x7fcf641a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:35.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf62a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf6419c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44760/0 (socket says 172.21.15.39:44760) 2024-09-21T01:26:35.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf62a00640 1 -- 172.21.15.39:0/950139572 learned_addr learned my addr 172.21.15.39:0/950139572 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:35.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf62000640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcf64076100 0x7fcf6419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:35.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf63400640 1 -- 172.21.15.39:0/950139572 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcf64076100 msgr2=0x7fcf6419ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf63400640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcf64076100 0x7fcf6419ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf63400640 1 -- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 msgr2=0x7fcf6419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf63400640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf6419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf63400640 1 -- 172.21.15.39:0/950139572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf641a2440 con 0x7fcf6410ff10 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.973+0000 7fcf62a00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf6419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.974+0000 7fcf63400640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6410ff10 0x7fcf641a1d10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00da10 tx=0x7fcf5c00ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.974+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00bd70 con 0x7fcf6410ff10 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.974+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf641a2730 con 0x7fcf6410ff10 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.974+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf5c003e20 con 0x7fcf6410ff10 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.974+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0106c0 con 0x7fcf6410ff10 2024-09-21T01:26:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.974+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641a8e90 con 0x7fcf6410ff10 2024-09-21T01:26:35.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.975+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf64075bc0 con 0x7fcf6410ff10 2024-09-21T01:26:35.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.978+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fcf5c004270 con 0x7fcf6410ff10 2024-09-21T01:26:35.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.06 MB/sec execute 422 sec latency 20171.704 ms 2024-09-21T01:26:35.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.979+0000 7fcf5b400640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcf480b6680 0x7fcf480b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:35.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.979+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7fcf5c0e8280 con 0x7fcf6410ff10 2024-09-21T01:26:35.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.980+0000 7fcf62a00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcf480b6680 0x7fcf480b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:35.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.980+0000 7fcf62a00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcf480b6680 0x7fcf480b8b90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcf54031440 tx=0x7fcf540058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:35.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:35.981+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcf5c0ae390 con 0x7fcf6410ff10 2024-09-21T01:26:36.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:35 smithi150 ceph-mon[34945]: pgmap v1115: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 11 op/s 2024-09-21T01:26:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:35 smithi160 ceph-mon[34696]: pgmap v1115: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 11 op/s 2024-09-21T01:26:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:35 smithi039 ceph-mon[30415]: pgmap v1115: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 11 op/s 2024-09-21T01:26:36.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.221+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf6419d600 con 0x7fcf6410ff10 2024-09-21T01:26:36.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.222+0000 7fcf5b400640 1 -- 172.21.15.39:0/950139572 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+11917 (secure 0 0 0) 0x7fcf5c0b2770 con 0x7fcf6410ff10 2024-09-21T01:26:36.222 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:26:36.222 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":61,"btime":"2024-09-21T01:25:54:810434+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":61,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:25:54.408578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:26:36.223 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 61 2024-09-21T01:26:36.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcf480b6680 msgr2=0x7fcf480b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:36.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcf480b6680 0x7fcf480b8b90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcf54031440 tx=0x7fcf540058d0 comp rx=0 tx=0).stop 2024-09-21T01:26:36.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6410ff10 msgr2=0x7fcf641a1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6410ff10 0x7fcf641a1d10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00da10 tx=0x7fcf5c00ded0 comp rx=0 tx=0).stop 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 shutdown_connections 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fcf480b6680 0x7fcf480b8b90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6410ff10 0x7fcf641a1d10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcf64076100 0x7fcf6419ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 --2- 172.21.15.39:0/950139572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf640757b0 0x7fcf6419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 >> 172.21.15.39:0/950139572 conn(0x7fcf640ff440 msgr2=0x7fcf64109ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 shutdown_connections 2024-09-21T01:26:36.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:36.224+0000 7fcf63e00640 1 -- 172.21.15.39:0/950139572 wait complete. 2024-09-21T01:26:36.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:36 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/950139572' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:26:36.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.03 MB/sec execute 423 sec latency 21171.766 ms 2024-09-21T01:26:37.052 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id a2eb49a5 --release --await' 2024-09-21T01:26:37.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id a2eb49a5 --release --await 2024-09-21T01:26:37.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:36 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/950139572' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:26:37.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:36 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/950139572' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:26:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:26:37.876 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:37 smithi039 ceph-mon[30415]: pgmap v1116: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 10 op/s 2024-09-21T01:26:37.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 145956 14.00 MB/sec execute 424 sec latency 22171.826 ms 2024-09-21T01:26:38.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:37 smithi150 ceph-mon[34945]: pgmap v1116: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 10 op/s 2024-09-21T01:26:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:37 smithi160 ceph-mon[34696]: pgmap v1116: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 10 op/s 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 -- 172.21.15.39:0/3325074161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f70105320 msgr2=0x7f1f7010d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 --2- 172.21.15.39:0/3325074161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f70105320 0x7f1f7010d870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009860 tx=0x7f1f5c02a340 comp rx=0 tx=0).stop 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 -- 172.21.15.39:0/3325074161 shutdown_connections 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 --2- 172.21.15.39:0/3325074161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f7010de30 0x7f1f70110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 --2- 172.21.15.39:0/3325074161 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f70105320 0x7f1f7010d870 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 --2- 172.21.15.39:0/3325074161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f701049d0 0x7f1f70104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 -- 172.21.15.39:0/3325074161 >> 172.21.15.39:0/3325074161 conn(0x7f1f70079740 msgr2=0x7f1f70079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 -- 172.21.15.39:0/3325074161 shutdown_connections 2024-09-21T01:26:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.195+0000 7f1f78000640 1 -- 172.21.15.39:0/3325074161 wait complete. 2024-09-21T01:26:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.197+0000 7f1f78000640 1 Processor -- start 2024-09-21T01:26:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.197+0000 7f1f78000640 1 -- start start 2024-09-21T01:26:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.197+0000 7f1f78000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f701049d0 0x7f1f7019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.197+0000 7f1f78000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70105320 0x7f1f7019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.197+0000 7f1f78000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f7010de30 0x7f1f7019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f78000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70113a90 con 0x7f1f7010de30 2024-09-21T01:26:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f78000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70113910 con 0x7f1f70105320 2024-09-21T01:26:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f78000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70113c10 con 0x7f1f701049d0 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f76000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f7010de30 0x7f1f7019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f75600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f701049d0 0x7f1f7019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f75600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f701049d0 0x7f1f7019a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59146/0 (socket says 172.21.15.39:59146) 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f75600640 1 -- 172.21.15.39:0/3077615179 learned_addr learned my addr 172.21.15.39:0/3077615179 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f74c00640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70105320 0x7f1f7019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f76000640 1 -- 172.21.15.39:0/3077615179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f701049d0 msgr2=0x7f1f7019a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f76000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f701049d0 0x7f1f7019a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f76000640 1 -- 172.21.15.39:0/3077615179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70105320 msgr2=0x7f1f7019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f76000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70105320 0x7f1f7019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.198+0000 7f1f76000640 1 -- 172.21.15.39:0/3077615179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f701a0470 con 0x7f1f7010de30 2024-09-21T01:26:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.199+0000 7f1f76000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f7010de30 0x7f1f7019fd40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800e970 tx=0x7f1f6800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:38.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.199+0000 7f1f6e000640 1 -- 172.21.15.39:0/3077615179 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800ccc0 con 0x7f1f7010de30 2024-09-21T01:26:38.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.199+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f701a0760 con 0x7f1f7010de30 2024-09-21T01:26:38.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.199+0000 7f1f6e000640 1 -- 172.21.15.39:0/3077615179 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f68005020 con 0x7f1f7010de30 2024-09-21T01:26:38.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.199+0000 7f1f6e000640 1 -- 172.21.15.39:0/3077615179 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68004a00 con 0x7f1f7010de30 2024-09-21T01:26:38.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.199+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f701a6ec0 con 0x7f1f7010de30 2024-09-21T01:26:38.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.202+0000 7f1f7ec6a740 1 Processor -- start 2024-09-21T01:26:38.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f6e000640 1 -- 172.21.15.39:0/3077615179 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f1f68004ba0 con 0x7f1f7010de30 2024-09-21T01:26:38.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 -- start start 2024-09-21T01:26:38.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b981c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b98700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b98c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x561572b0f290 con 0x5615727d0840 2024-09-21T01:26:38.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561572afe6c0 con 0x5615727c8660 2024-09-21T01:26:38.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.203+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x561572b024d0 con 0x5615727f1520 2024-09-21T01:26:38.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59154/0 (socket says 172.21.15.39:59154) 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 -- 172.21.15.39:0/116715961 learned_addr learned my addr 172.21.15.39:0/116715961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f75600640 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b98700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b98c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 -- 172.21.15.39:0/116715961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 msgr2=0x561572b98700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b98700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 -- 172.21.15.39:0/116715961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 msgr2=0x561572b98c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b98c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.204+0000 7f1f74c00640 1 -- 172.21.15.39:0/116715961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561572a8b460 con 0x5615727f1520 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f74c00640 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b981c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c02a930 tx=0x7f1f5c005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f4445da399ee6c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f6e000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1f580b6680 0x7f1f580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f6e000640 1 -- 172.21.15.39:0/3077615179 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7f1f680e7a80 con 0x7f1f7010de30 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f62600640 1 -- 172.21.15.39:0/116715961 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c037040 con 0x5615727f1520 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f62600640 1 -- 172.21.15.39:0/116715961 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f5c030450 con 0x5615727f1520 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f75600640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1f580b6680 0x7f1f580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.205+0000 7f1f62600640 1 -- 172.21.15.39:0/116715961 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0037f0 con 0x5615727f1520 2024-09-21T01:26:38.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f75600640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1f580b6680 0x7f1f580b8b90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x561572b96e40 tx=0x7f1f64010040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:38.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/116715961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 msgr2=0x561572b981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b981c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c02a930 tx=0x7f1f5c005780 comp rx=0 tx=0).stop 2024-09-21T01:26:38.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/116715961 shutdown_connections 2024-09-21T01:26:38.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b98c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b98700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/116715961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b981c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/116715961 >> 172.21.15.39:0/116715961 conn(0x5615727e5910 msgr2=0x561572b960b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:38.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/116715961 shutdown_connections 2024-09-21T01:26:38.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.206+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/116715961 wait complete. 2024-09-21T01:26:38.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.207+0000 7f1f7ec6a740 1 -- start start 2024-09-21T01:26:38.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.207+0000 7f1f7ec6a740 1 Processor -- start 2024-09-21T01:26:38.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 0x561572ba3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727f1520 0x561572bafc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56157295fbb0 con 0x5615727c8660 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561572a996c0 con 0x5615727f1520 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x561572ba6940 con 0x5615727d0840 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f7ec6a740 10 client.0 _ll_register_callbacks cb 0x561572ab4490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 0x561572ba3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 0x561572ba3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44790/0 (socket says 172.21.15.39:44790) 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 -- 172.21.15.39:0/1666885647 learned_addr learned my addr 172.21.15.39:0/1666885647 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f74c00640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f76000640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727f1520 0x561572bafc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 -- 172.21.15.39:0/1666885647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 msgr2=0x561572ba4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 -- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727f1520 msgr2=0x561572bafc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727f1520 0x561572bafc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f75600640 1 -- 172.21.15.39:0/1666885647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56157285cb00 con 0x5615727c8660 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f74c00640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.208+0000 7f1f76000640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727f1520 0x561572bafc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.209+0000 7f1f75600640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 0x561572ba3d40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400ffd0 tx=0x7f1f64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822c403145a5f23b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.209+0000 7f1f56a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.209+0000 7f1f56a00640 1 -- 172.21.15.39:0/1666885647 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64018630 con 0x5615727c8660 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.209+0000 7f1f56a00640 1 -- 172.21.15.39:0/1666885647 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7f1f640187d0 con 0x5615727c8660 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.209+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f5c030450 con 0x5615727c8660 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f56a00640 1 -- 172.21.15.39:0/1666885647 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f1f64033050 con 0x5615727c8660 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 10 client.16452 fetch_fsmap learned FSMap version 61 2024-09-21T01:26:38.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x5615728d3960 con 0x5615727c8660 2024-09-21T01:26:38.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f56a00640 1 -- 172.21.15.39:0/1666885647 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f5c030450 con 0x5615727c8660 2024-09-21T01:26:38.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f56a00640 1 -- 172.21.15.39:0/1666885647 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 61) ==== 8370+0+0 (secure 0 0 0) 0x7f1f64024070 con 0x5615727c8660 2024-09-21T01:26:38.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 10 client.16452 fetch_fsmap finished waiting for FSMap version 61 2024-09-21T01:26:38.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 10 client.16452 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 20 client.16452 populate_metadata read hostname 'smithi039' 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5615727afea0 0x561572bb5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 4 client.16452 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.210+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561572b111e0 con 0x5615727afea0 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.211+0000 7f1f74c00640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5615727afea0 0x561572bb5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.211+0000 7f1f74c00640 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5615727afea0 0x561572bb5da0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a7764a33f10b5f server_cookie=3dc67be7ecc84bba in_seq=0 out_seq=0 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.211+0000 7f1f56a00640 10 client.16452 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.212+0000 7f1f56a00640 1 -- 172.21.15.39:0/1666885647 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561572b111e0 con 0x5615727afea0 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.212+0000 7f1f56a00640 10 client.16452 handle_command_reply: tid=0 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 client.16452 shutdown 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 msgr2=0x561572ba3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 0x561572ba3d40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400ffd0 tx=0x7f1f64004ab0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 shutdown_connections 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x5615727afea0 0x561572bb5da0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727f1520 0x561572bafc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/1666885647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727c8660 0x561572ba3d40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 >> 172.21.15.39:0/1666885647 conn(0x5615727e5910 msgr2=0x561572b9b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f56a00640 0 client.16452 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 shutdown_connections 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/1666885647 wait complete. 2024-09-21T01:26:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.213+0000 7f1f7ec6a740 20 client.16452 trim_cache size 0 max 16384 2024-09-21T01:26:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 Processor -- start 2024-09-21T01:26:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 -- start start 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b99bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b95660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b95ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1f701b39d0 con 0x5615727d0840 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1f701b1240 con 0x5615727c8660 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f701af0b0 con 0x5615727f1520 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f74c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b95660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f75600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b99bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f74c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b95660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44806/0 (socket says 172.21.15.39:44806) 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.232+0000 7f1f74c00640 1 -- 172.21.15.39:0/521146470 learned_addr learned my addr 172.21.15.39:0/521146470 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f76000640 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b95ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f75600640 1 -- 172.21.15.39:0/521146470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 msgr2=0x561572b95ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f75600640 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b95ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f75600640 1 -- 172.21.15.39:0/521146470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 msgr2=0x561572b95660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f75600640 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b95660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f75600640 1 -- 172.21.15.39:0/521146470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561572b9a9c0 con 0x5615727c8660 2024-09-21T01:26:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f76000640 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b95ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f75600640 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b99bb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009a60 tx=0x7f1f6400ff00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a07eb50a3b4724b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f61c00640 1 -- 172.21.15.39:0/521146470 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64025430 con 0x5615727c8660 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f61c00640 1 -- 172.21.15.39:0/521146470 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f6403c070 con 0x5615727c8660 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.233+0000 7f1f61c00640 1 -- 172.21.15.39:0/521146470 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64018e30 con 0x5615727c8660 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/521146470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 msgr2=0x561572b99bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b99bb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009a60 tx=0x7f1f6400ff00 comp rx=0 tx=0).stop 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/521146470 shutdown_connections 2024-09-21T01:26:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727f1520 0x561572b95ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727d0840 0x561572b95660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/521146470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572b99bb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/521146470 >> 172.21.15.39:0/521146470 conn(0x5615727e5910 msgr2=0x561572b9b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/521146470 shutdown_connections 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.234+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/521146470 wait complete. 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.235+0000 7f1f7ec6a740 1 -- start start 2024-09-21T01:26:38.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.235+0000 7f1f7ec6a740 1 Processor -- start 2024-09-21T01:26:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572ba7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 0x561572c23bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1f701af700 con 0x5615727f1520 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1f701b0b70 con 0x5615727c8660 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1f701afdd0 con 0x5615727d0840 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f7ec6a740 10 client.0 _ll_register_callbacks cb 0x7f1f64009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 0x561572c23bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 0x561572c23bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44818/0 (socket says 172.21.15.39:44818) 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 -- 172.21.15.39:0/3477954773 learned_addr learned my addr 172.21.15.39:0/3477954773 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f74c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f75600640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572ba7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 -- 172.21.15.39:0/3477954773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 msgr2=0x561572ba8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 -- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 msgr2=0x561572ba7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:26:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572ba7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f76000640 1 -- 172.21.15.39:0/3477954773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561572ba8b40 con 0x5615727f1520 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f74c00640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.236+0000 7f1f75600640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572ba7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f76000640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 0x561572c23bb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800e940 tx=0x7f1f680acf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2104c7907d814670 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f56a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f56a00640 1 -- 172.21.15.39:0/3477954773 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f680d4070 con 0x5615727f1520 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f56a00640 1 -- 172.21.15.39:0/3477954773 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9511+0+0 (secure 0 0 0) 0x7f1f680985f0 con 0x5615727f1520 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f6403c070 con 0x5615727f1520 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f56a00640 1 -- 172.21.15.39:0/3477954773 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f1f680c8050 con 0x5615727f1520 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f7ec6a740 10 client.16458 fetch_fsmap learned FSMap version 61 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.237+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x561572c250f0 con 0x5615727f1520 2024-09-21T01:26:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f56a00640 1 -- 172.21.15.39:0/3477954773 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f6403c070 con 0x5615727f1520 2024-09-21T01:26:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f56a00640 1 -- 172.21.15.39:0/3477954773 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 61) ==== 8370+0+0 (secure 0 0 0) 0x7f1f680b6040 con 0x5615727f1520 2024-09-21T01:26:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f7ec6a740 10 client.16458 fetch_fsmap finished waiting for FSMap version 61 2024-09-21T01:26:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f7ec6a740 10 client.16458 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:26:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f7ec6a740 20 client.16458 populate_metadata read hostname 'smithi039' 2024-09-21T01:26:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x561572785250 0x561572c27910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f7ec6a740 4 client.16458 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a2eb49a5", "release": true, "await": true}] 2024-09-21T01:26:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a2eb49a5", "release": true, "await": true}) -- 0x561572b0cc90 con 0x561572785250 2024-09-21T01:26:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.238+0000 7f1f75600640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x561572785250 0x561572c27910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.239+0000 7f1f75600640 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x561572785250 0x561572c27910 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32a3aecce5c42580 server_cookie=5abc169df9f2c757 in_seq=0 out_seq=0 2024-09-21T01:26:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.239+0000 7f1f56a00640 10 client.16458 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.277+0000 7f1f56a00640 1 -- 172.21.15.39:0/3477954773 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561572b0cc90 con 0x561572785250 2024-09-21T01:26:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.277+0000 7f1f56a00640 10 client.16458 handle_command_reply: tid=0 2024-09-21T01:26:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.277+0000 7f1f7ec6a740 1 client.16458 shutdown 2024-09-21T01:26:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.279+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 msgr2=0x561572c23bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.279+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 0x561572c23bb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800e940 tx=0x7f1f680acf10 comp rx=0 tx=0).stop 2024-09-21T01:26:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 shutdown_connections 2024-09-21T01:26:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x561572785250 0x561572c27910 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5615727f1520 0x561572c23bb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x5615727d0840 0x561572ba8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 --2- 172.21.15.39:0/3477954773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615727c8660 0x561572ba7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 >> 172.21.15.39:0/3477954773 conn(0x5615727e5910 msgr2=0x561572ba23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f56a00640 0 client.16458 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.280+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 shutdown_connections 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.281+0000 7f1f7ec6a740 1 -- 172.21.15.39:0/3477954773 wait complete. 2024-09-21T01:26:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.281+0000 7f1f7ec6a740 20 client.16458 trim_cache size 0 max 16384 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 61, 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 88, 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "a2eb49a5": { 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 88, 2024-09-21T01:26:38.282 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 149.8, 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:38.283 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:38.284 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:38.284 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:26:38.284 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:26:38.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.282+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1f580b6680 msgr2=0x7f1f580b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.282+0000 7f1f78000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1f580b6680 0x7f1f580b8b90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x561572b96e40 tx=0x7f1f64010040 comp rx=0 tx=0).stop 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.282+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f7010de30 msgr2=0x7f1f7019fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.282+0000 7f1f78000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f7010de30 0x7f1f7019fd40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800e970 tx=0x7f1f6800ee30 comp rx=0 tx=0).stop 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 shutdown_connections 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f7010de30 0x7f1f7019fd40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1f580b6680 0x7f1f580b8b90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70105320 0x7f1f7019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 --2- 172.21.15.39:0/3077615179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1f701049d0 0x7f1f7019a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 >> 172.21.15.39:0/3077615179 conn(0x7f1f70079740 msgr2=0x7f1f701007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 shutdown_connections 2024-09-21T01:26:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:26:38.283+0000 7f1f78000640 1 -- 172.21.15.39:0/3077615179 wait complete. 2024-09-21T01:26:38.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 146330 14.00 MB/sec execute 425 sec latency 22479.638 ms 2024-09-21T01:26:39.077 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a2eb49a5', total seconds elapsed: 34.0 2024-09-21T01:26:39.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 146912 14.04 MB/sec execute 426 sec latency 106.854 ms 2024-09-21T01:26:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:39 smithi150 ceph-mon[34945]: pgmap v1117: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 233 KiB/s wr, 10 op/s 2024-09-21T01:26:40.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:39 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:39 smithi160 ceph-mon[34696]: pgmap v1117: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 233 KiB/s wr, 10 op/s 2024-09-21T01:26:40.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:39 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:40.164 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:39 smithi039 ceph-mon[30415]: pgmap v1117: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 233 KiB/s wr, 10 op/s 2024-09-21T01:26:40.165 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:39 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:40.980 INFO:tasks.workunit.client.0.smithi039.stdout: 10 147364 14.07 MB/sec execute 427 sec latency 183.765 ms 2024-09-21T01:26:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:41 smithi039 ceph-mon[30415]: pgmap v1118: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-09-21T01:26:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:41.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:41.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:26:41.981 INFO:tasks.workunit.client.0.smithi039.stdout: 10 147706 14.08 MB/sec execute 428 sec latency 316.633 ms 2024-09-21T01:26:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:41 smithi150 ceph-mon[34945]: pgmap v1118: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-09-21T01:26:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:42.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:26:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:41 smithi160 ceph-mon[34696]: pgmap v1118: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-09-21T01:26:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:42.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:26:42.981 INFO:tasks.workunit.client.0.smithi039.stdout: 10 148088 14.08 MB/sec execute 429 sec latency 413.482 ms 2024-09-21T01:26:43.982 INFO:tasks.workunit.client.0.smithi039.stdout: 10 148715 14.14 MB/sec execute 430 sec latency 205.300 ms 2024-09-21T01:26:44.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:43 smithi150 ceph-mon[34945]: pgmap v1119: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 89 op/s 2024-09-21T01:26:44.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:43 smithi160 ceph-mon[34696]: pgmap v1119: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 89 op/s 2024-09-21T01:26:44.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:43 smithi039 ceph-mon[30415]: pgmap v1119: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 89 op/s 2024-09-21T01:26:44.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 149083 14.17 MB/sec execute 431 sec latency 358.751 ms 2024-09-21T01:26:45.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 149483 14.18 MB/sec execute 432 sec latency 141.004 ms 2024-09-21T01:26:46.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:45 smithi150 ceph-mon[34945]: pgmap v1120: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 164 op/s 2024-09-21T01:26:46.074 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:45 smithi039 ceph-mon[30415]: pgmap v1120: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 164 op/s 2024-09-21T01:26:46.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:45 smithi160 ceph-mon[34696]: pgmap v1120: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 164 op/s 2024-09-21T01:26:46.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 149884 14.18 MB/sec execute 433 sec latency 141.522 ms 2024-09-21T01:26:47.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 150439 14.22 MB/sec execute 434 sec latency 355.184 ms 2024-09-21T01:26:48.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:47 smithi150 ceph-mon[34945]: pgmap v1121: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.7 MiB/s wr, 246 op/s 2024-09-21T01:26:48.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:47 smithi160 ceph-mon[34696]: pgmap v1121: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.7 MiB/s wr, 246 op/s 2024-09-21T01:26:48.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:47 smithi039 ceph-mon[30415]: pgmap v1121: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.7 MiB/s wr, 246 op/s 2024-09-21T01:26:48.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 150961 14.25 MB/sec execute 435 sec latency 1355.239 ms 2024-09-21T01:26:49.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:48 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:26:49.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:48 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:26:49.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:48 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:26:49.985 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:49 smithi039 ceph-mon[30415]: pgmap v1122: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.0 MiB/s wr, 324 op/s 2024-09-21T01:26:49.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 151617 14.31 MB/sec execute 436 sec latency 2355.291 ms 2024-09-21T01:26:50.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:49 smithi150 ceph-mon[34945]: pgmap v1122: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.0 MiB/s wr, 324 op/s 2024-09-21T01:26:50.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:49 smithi160 ceph-mon[34696]: pgmap v1122: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.0 MiB/s wr, 324 op/s 2024-09-21T01:26:50.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 152228 14.35 MB/sec execute 437 sec latency 3355.354 ms 2024-09-21T01:26:51.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 152735 14.39 MB/sec execute 438 sec latency 4355.407 ms 2024-09-21T01:26:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:51 smithi150 ceph-mon[34945]: pgmap v1123: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 431 op/s 2024-09-21T01:26:52.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:51 smithi160 ceph-mon[34696]: pgmap v1123: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 431 op/s 2024-09-21T01:26:52.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:51 smithi039 ceph-mon[30415]: pgmap v1123: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 431 op/s 2024-09-21T01:26:52.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 153118 14.39 MB/sec execute 439 sec latency 5355.469 ms 2024-09-21T01:26:53.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 153538 14.42 MB/sec execute 440 sec latency 6355.531 ms 2024-09-21T01:26:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:53 smithi150 ceph-mon[34945]: pgmap v1124: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.6 MiB/s wr, 417 op/s 2024-09-21T01:26:54.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:53 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:26:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:53 smithi160 ceph-mon[34696]: pgmap v1124: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.6 MiB/s wr, 417 op/s 2024-09-21T01:26:54.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:53 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:26:54.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:53 smithi039 ceph-mon[30415]: pgmap v1124: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.6 MiB/s wr, 417 op/s 2024-09-21T01:26:54.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:53 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:26:54.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 154028 14.43 MB/sec execute 441 sec latency 6854.122 ms 2024-09-21T01:26:55.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 154623 14.48 MB/sec execute 442 sec latency 218.665 ms 2024-09-21T01:26:56.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:55 smithi150 ceph-mon[34945]: pgmap v1125: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 426 op/s 2024-09-21T01:26:56.099 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:55 smithi039 ceph-mon[30415]: pgmap v1125: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 426 op/s 2024-09-21T01:26:56.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:55 smithi160 ceph-mon[34696]: pgmap v1125: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 426 op/s 2024-09-21T01:26:56.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 155226 14.52 MB/sec execute 443 sec latency 77.199 ms 2024-09-21T01:26:57.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 155612 14.54 MB/sec execute 444 sec latency 256.924 ms 2024-09-21T01:26:58.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:57 smithi160 ceph-mon[34696]: pgmap v1126: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 422 op/s 2024-09-21T01:26:58.146 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:57 smithi039 ceph-mon[30415]: pgmap v1126: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 422 op/s 2024-09-21T01:26:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:57 smithi150 ceph-mon[34945]: pgmap v1126: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 422 op/s 2024-09-21T01:26:58.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156005 14.55 MB/sec execute 445 sec latency 194.782 ms 2024-09-21T01:26:59.986 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156551 14.58 MB/sec execute 446 sec latency 107.357 ms 2024-09-21T01:27:00.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:26:59 smithi160 ceph-mon[34696]: pgmap v1127: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.1 MiB/s wr, 429 op/s 2024-09-21T01:27:00.156 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:26:59 smithi039 ceph-mon[30415]: pgmap v1127: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.1 MiB/s wr, 429 op/s 2024-09-21T01:27:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:26:59 smithi150 ceph-mon[34945]: pgmap v1127: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.1 MiB/s wr, 429 op/s 2024-09-21T01:27:01.006 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156648 14.56 MB/sec execute 447 sec latency 843.947 ms 2024-09-21T01:27:01.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:01 smithi039 ceph-mon[30415]: pgmap v1128: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.0 MiB/s wr, 437 op/s 2024-09-21T01:27:02.010 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156651 14.53 MB/sec execute 448 sec latency 1847.927 ms 2024-09-21T01:27:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:01 smithi150 ceph-mon[34945]: pgmap v1128: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.0 MiB/s wr, 437 op/s 2024-09-21T01:27:02.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:01 smithi160 ceph-mon[34696]: pgmap v1128: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.0 MiB/s wr, 437 op/s 2024-09-21T01:27:03.010 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156651 14.50 MB/sec execute 449 sec latency 2847.979 ms 2024-09-21T01:27:04.010 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156653 14.47 MB/sec execute 450 sec latency 3848.022 ms 2024-09-21T01:27:04.134 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:03 smithi039 ceph-mon[30415]: pgmap v1129: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 7.9 MiB/s wr, 421 op/s 2024-09-21T01:27:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:03 smithi150 ceph-mon[34945]: pgmap v1129: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 7.9 MiB/s wr, 421 op/s 2024-09-21T01:27:04.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:03 smithi160 ceph-mon[34696]: pgmap v1129: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 7.9 MiB/s wr, 421 op/s 2024-09-21T01:27:05.010 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156735 14.45 MB/sec execute 451 sec latency 4688.850 ms 2024-09-21T01:27:05.965 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:05 smithi039 ceph-mon[30415]: pgmap v1130: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.5 MiB/s wr, 425 op/s 2024-09-21T01:27:05.965 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:05 smithi039 ceph-mon[30415]: osdmap e125: 12 total, 12 up, 12 in 2024-09-21T01:27:06.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156788 14.42 MB/sec execute 452 sec latency 4861.229 ms 2024-09-21T01:27:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:05 smithi150 ceph-mon[34945]: pgmap v1130: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.5 MiB/s wr, 425 op/s 2024-09-21T01:27:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:05 smithi150 ceph-mon[34945]: osdmap e125: 12 total, 12 up, 12 in 2024-09-21T01:27:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:05 smithi160 ceph-mon[34696]: pgmap v1130: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.5 MiB/s wr, 425 op/s 2024-09-21T01:27:06.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:05 smithi160 ceph-mon[34696]: osdmap e125: 12 total, 12 up, 12 in 2024-09-21T01:27:07.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156789 14.39 MB/sec execute 453 sec latency 1775.879 ms 2024-09-21T01:27:07.974 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:07 smithi039 ceph-mon[30415]: pgmap v1132: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.6 MiB/s wr, 433 op/s 2024-09-21T01:27:08.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156833 14.36 MB/sec execute 454 sec latency 2775.928 ms 2024-09-21T01:27:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:07 smithi150 ceph-mon[34945]: pgmap v1132: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.6 MiB/s wr, 433 op/s 2024-09-21T01:27:08.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:07 smithi160 ceph-mon[34696]: pgmap v1132: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.6 MiB/s wr, 433 op/s 2024-09-21T01:27:09.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156854 14.33 MB/sec execute 455 sec latency 3775.977 ms 2024-09-21T01:27:10.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156854 14.30 MB/sec execute 456 sec latency 4776.021 ms 2024-09-21T01:27:10.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:09 smithi039 ceph-mon[30415]: pgmap v1133: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.6 MiB/s wr, 389 op/s 2024-09-21T01:27:10.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:09 smithi150 ceph-mon[34945]: pgmap v1133: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.6 MiB/s wr, 389 op/s 2024-09-21T01:27:10.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:09 smithi160 ceph-mon[34696]: pgmap v1133: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.6 MiB/s wr, 389 op/s 2024-09-21T01:27:11.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 156926 14.27 MB/sec execute 457 sec latency 5776.064 ms 2024-09-21T01:27:12.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 157212 14.27 MB/sec execute 458 sec latency 6444.133 ms 2024-09-21T01:27:12.072 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:11 smithi039 ceph-mon[30415]: pgmap v1134: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.7 MiB/s wr, 357 op/s 2024-09-21T01:27:12.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:11 smithi150 ceph-mon[34945]: pgmap v1134: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.7 MiB/s wr, 357 op/s 2024-09-21T01:27:12.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:11 smithi160 ceph-mon[34696]: pgmap v1134: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.7 MiB/s wr, 357 op/s 2024-09-21T01:27:13.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 157669 14.27 MB/sec execute 459 sec latency 139.631 ms 2024-09-21T01:27:14.023 INFO:tasks.workunit.client.0.smithi039.stdout: 10 158478 14.35 MB/sec execute 460 sec latency 107.665 ms 2024-09-21T01:27:14.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:13 smithi039 ceph-mon[30415]: pgmap v1135: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.4 MiB/s wr, 284 op/s 2024-09-21T01:27:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:13 smithi150 ceph-mon[34945]: pgmap v1135: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.4 MiB/s wr, 284 op/s 2024-09-21T01:27:14.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:13 smithi160 ceph-mon[34696]: pgmap v1135: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.4 MiB/s wr, 284 op/s 2024-09-21T01:27:15.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 159028 14.40 MB/sec execute 461 sec latency 177.748 ms 2024-09-21T01:27:15.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:15 smithi039 ceph-mon[30415]: pgmap v1136: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.6 MiB/s wr, 265 op/s 2024-09-21T01:27:15.905 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:15 smithi039 ceph-mon[30415]: osdmap e126: 12 total, 12 up, 12 in 2024-09-21T01:27:16.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 159525 14.43 MB/sec execute 462 sec latency 106.751 ms 2024-09-21T01:27:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:15 smithi150 ceph-mon[34945]: pgmap v1136: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.6 MiB/s wr, 265 op/s 2024-09-21T01:27:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:15 smithi150 ceph-mon[34945]: osdmap e126: 12 total, 12 up, 12 in 2024-09-21T01:27:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:15 smithi160 ceph-mon[34696]: pgmap v1136: 129 pgs: 129 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.6 MiB/s wr, 265 op/s 2024-09-21T01:27:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:15 smithi160 ceph-mon[34696]: osdmap e126: 12 total, 12 up, 12 in 2024-09-21T01:27:17.026 INFO:tasks.workunit.client.0.smithi039.stdout: 10 159956 14.45 MB/sec execute 463 sec latency 191.786 ms 2024-09-21T01:27:18.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 160463 14.46 MB/sec execute 464 sec latency 152.387 ms 2024-09-21T01:27:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:17 smithi150 ceph-mon[34945]: pgmap v1138: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.6 MiB/s wr, 271 op/s 2024-09-21T01:27:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:17 smithi160 ceph-mon[34696]: pgmap v1138: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.6 MiB/s wr, 271 op/s 2024-09-21T01:27:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:17 smithi039 ceph-mon[30415]: pgmap v1138: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.6 MiB/s wr, 271 op/s 2024-09-21T01:27:19.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 161086 14.50 MB/sec execute 465 sec latency 81.603 ms 2024-09-21T01:27:20.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 161754 14.54 MB/sec execute 466 sec latency 112.302 ms 2024-09-21T01:27:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:19 smithi150 ceph-mon[34945]: pgmap v1139: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.8 MiB/s wr, 318 op/s 2024-09-21T01:27:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:19 smithi160 ceph-mon[34696]: pgmap v1139: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.8 MiB/s wr, 318 op/s 2024-09-21T01:27:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:19 smithi039 ceph-mon[30415]: pgmap v1139: 129 pgs: 129 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.8 MiB/s wr, 318 op/s 2024-09-21T01:27:21.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 162460 14.61 MB/sec execute 467 sec latency 124.978 ms 2024-09-21T01:27:22.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 162829 14.63 MB/sec execute 468 sec latency 96.130 ms 2024-09-21T01:27:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:21 smithi150 ceph-mon[34945]: pgmap v1140: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.6 MiB/s wr, 359 op/s 2024-09-21T01:27:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:21 smithi160 ceph-mon[34696]: pgmap v1140: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.6 MiB/s wr, 359 op/s 2024-09-21T01:27:22.113 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:21 smithi039 ceph-mon[30415]: pgmap v1140: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 6.6 MiB/s wr, 359 op/s 2024-09-21T01:27:23.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 163291 14.64 MB/sec execute 469 sec latency 84.895 ms 2024-09-21T01:27:23.878 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:23 smithi039 ceph-mon[30415]: pgmap v1141: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 356 op/s 2024-09-21T01:27:24.024 INFO:tasks.workunit.client.0.smithi039.stdout: 10 163696 14.65 MB/sec execute 470 sec latency 104.219 ms 2024-09-21T01:27:24.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:23 smithi150 ceph-mon[34945]: pgmap v1141: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 356 op/s 2024-09-21T01:27:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:23 smithi160 ceph-mon[34696]: pgmap v1141: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 356 op/s 2024-09-21T01:27:25.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 164234 14.68 MB/sec execute 471 sec latency 155.354 ms 2024-09-21T01:27:26.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 164899 14.71 MB/sec execute 472 sec latency 115.894 ms 2024-09-21T01:27:26.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:25 smithi150 ceph-mon[34945]: pgmap v1142: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 390 op/s 2024-09-21T01:27:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:25 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:27:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:25 smithi160 ceph-mon[34696]: pgmap v1142: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 390 op/s 2024-09-21T01:27:26.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:25 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:27:26.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:25 smithi039 ceph-mon[30415]: pgmap v1142: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.9 MiB/s wr, 390 op/s 2024-09-21T01:27:26.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:25 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:27:27.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 165637 14.78 MB/sec execute 473 sec latency 88.792 ms 2024-09-21T01:27:28.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 166241 14.82 MB/sec execute 474 sec latency 111.952 ms 2024-09-21T01:27:28.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:27 smithi150 ceph-mon[34945]: pgmap v1143: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 390 op/s 2024-09-21T01:27:28.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:27 smithi039 ceph-mon[30415]: pgmap v1143: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 390 op/s 2024-09-21T01:27:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:27 smithi160 ceph-mon[34696]: pgmap v1143: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 390 op/s 2024-09-21T01:27:29.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 166745 14.85 MB/sec execute 475 sec latency 290.863 ms 2024-09-21T01:27:29.902 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:29 smithi039 ceph-mon[30415]: pgmap v1144: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 415 op/s 2024-09-21T01:27:30.025 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167196 14.87 MB/sec execute 476 sec latency 126.384 ms 2024-09-21T01:27:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:29 smithi150 ceph-mon[34945]: pgmap v1144: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 415 op/s 2024-09-21T01:27:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:29 smithi160 ceph-mon[34696]: pgmap v1144: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 415 op/s 2024-09-21T01:27:31.042 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167506 14.87 MB/sec execute 477 sec latency 390.230 ms 2024-09-21T01:27:32.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:31 smithi039 ceph-mon[30415]: pgmap v1145: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 407 op/s 2024-09-21T01:27:32.042 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167510 14.84 MB/sec execute 478 sec latency 1154.937 ms 2024-09-21T01:27:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:31 smithi150 ceph-mon[34945]: pgmap v1145: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 407 op/s 2024-09-21T01:27:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:31 smithi160 ceph-mon[34696]: pgmap v1145: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 407 op/s 2024-09-21T01:27:33.042 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167512 14.80 MB/sec execute 479 sec latency 2155.006 ms 2024-09-21T01:27:34.042 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167550 14.78 MB/sec execute 480 sec latency 3086.553 ms 2024-09-21T01:27:34.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:33 smithi150 ceph-mon[34945]: pgmap v1146: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 376 op/s 2024-09-21T01:27:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:33 smithi160 ceph-mon[34696]: pgmap v1146: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 376 op/s 2024-09-21T01:27:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:33 smithi039 ceph-mon[30415]: pgmap v1146: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 376 op/s 2024-09-21T01:27:34.420 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.9 2024-09-21T01:27:34.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:27:35.042 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167586 14.75 MB/sec execute 481 sec latency 4086.614 ms 2024-09-21T01:27:35.852 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:27:36.043 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167955 14.76 MB/sec execute 482 sec latency 4440.900 ms 2024-09-21T01:27:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:35 smithi160 ceph-mon[34696]: pgmap v1147: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 384 op/s 2024-09-21T01:27:36.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:35 smithi039 ceph-mon[30415]: pgmap v1147: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 384 op/s 2024-09-21T01:27:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:35 smithi150 ceph-mon[34945]: pgmap v1147: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.2 MiB/s wr, 384 op/s 2024-09-21T01:27:37.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167962 14.73 MB/sec execute 483 sec latency 1036.558 ms 2024-09-21T01:27:37.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.724+0000 7f09ca600640 1 -- 172.21.15.39:0/2492235845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c41149c0 msgr2=0x7f09c4116de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:37.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.728+0000 7f09ca600640 1 --2- 172.21.15.39:0/2492235845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c41149c0 0x7f09c4116de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005fd0 tx=0x7f09bc02b3f0 comp rx=0 tx=0).stop 2024-09-21T01:27:37.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.729+0000 7f09ca600640 1 -- 172.21.15.39:0/2492235845 shutdown_connections 2024-09-21T01:27:37.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.729+0000 7f09ca600640 1 --2- 172.21.15.39:0/2492235845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c41149c0 0x7f09c4116de0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:37.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.729+0000 7f09ca600640 1 --2- 172.21.15.39:0/2492235845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c410bb90 0x7f09c410bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:37.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.729+0000 7f09ca600640 1 --2- 172.21.15.39:0/2492235845 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c410b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:37.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.729+0000 7f09ca600640 1 -- 172.21.15.39:0/2492235845 >> 172.21.15.39:0/2492235845 conn(0x7f09c406e9f0 msgr2=0x7f09c406ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:37.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.730+0000 7f09ca600640 1 -- 172.21.15.39:0/2492235845 shutdown_connections 2024-09-21T01:27:37.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.730+0000 7f09ca600640 1 -- 172.21.15.39:0/2492235845 wait complete. 2024-09-21T01:27:37.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.732+0000 7f09ca600640 1 Processor -- start 2024-09-21T01:27:37.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.732+0000 7f09ca600640 1 -- start start 2024-09-21T01:27:37.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09ca600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c411b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:37.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09ca600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c410bb90 0x7f09c411be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:37.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09ca600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c41149c0 0x7f09c411c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:37.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09ca600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f09c4119b30 con 0x7f09c41149c0 2024-09-21T01:27:37.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09ca600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f09c41199b0 con 0x7f09c410bb90 2024-09-21T01:27:37.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09ca600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f09c4119cb0 con 0x7f09c410ab30 2024-09-21T01:27:37.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c411b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c411b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:33462/0 (socket says 172.21.15.39:33462) 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 -- 172.21.15.39:0/816945566 learned_addr learned my addr 172.21.15.39:0/816945566 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c3e00640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c410bb90 0x7f09c411be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 -- 172.21.15.39:0/816945566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c410bb90 msgr2=0x7f09c411be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c410bb90 0x7f09c411be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 -- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c41149c0 msgr2=0x7f09c411c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c41149c0 0x7f09c411c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.738+0000 7f09c9200640 1 -- 172.21.15.39:0/816945566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c41b50e0 con 0x7f09c410ab30 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.743+0000 7f09c3e00640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c410bb90 0x7f09c411be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:27:37.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.744+0000 7f09c9200640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c411b950 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09b800d950 tx=0x7f09b800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:37.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.745+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8017070 con 0x7f09c410ab30 2024-09-21T01:27:37.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.745+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09b8009c30 con 0x7f09c410ab30 2024-09-21T01:27:37.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.745+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b800f620 con 0x7f09c410ab30 2024-09-21T01:27:37.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.745+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c41b53d0 con 0x7f09c410ab30 2024-09-21T01:27:37.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.746+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c41b5960 con 0x7f09c410ab30 2024-09-21T01:27:37.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.748+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c410b650 con 0x7f09c410ab30 2024-09-21T01:27:37.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.751+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f09b8010460 con 0x7f09c410ab30 2024-09-21T01:27:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.757+0000 7f09c1600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09ac0b6780 0x7f09ac0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.757+0000 7f09c3e00640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09ac0b6780 0x7f09ac0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.757+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7f09b80e7b40 con 0x7f09c410ab30 2024-09-21T01:27:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.757+0000 7f09c3e00640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09ac0b6780 0x7f09ac0b8c90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09c4117130 tx=0x7f09b0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:37.757+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f09b8020070 con 0x7f09c410ab30 2024-09-21T01:27:38.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167965 14.70 MB/sec execute 484 sec latency 2036.637 ms 2024-09-21T01:27:38.090 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:37 smithi039 ceph-mon[30415]: pgmap v1148: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 363 op/s 2024-09-21T01:27:38.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:37 smithi160 ceph-mon[34696]: pgmap v1148: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 363 op/s 2024-09-21T01:27:38.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.213+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09c411cc50 con 0x7f09c410ab30 2024-09-21T01:27:38.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.213+0000 7f09c1600640 1 -- 172.21.15.39:0/816945566 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11915 (secure 0 0 0) 0x7f09b80adcd0 con 0x7f09c410ab30 2024-09-21T01:27:38.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:27:38.220 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":64,"btime":"2024-09-21T01:27:24:747931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:27:24.577857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:27:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 64 2024-09-21T01:27:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.219+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09ac0b6780 msgr2=0x7f09ac0b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.219+0000 7f09ca600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09ac0b6780 0x7f09ac0b8c90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09c4117130 tx=0x7f09b0005f90 comp rx=0 tx=0).stop 2024-09-21T01:27:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.219+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 msgr2=0x7f09c411b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.219+0000 7f09ca600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c411b950 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09b800d950 tx=0x7f09b800de10 comp rx=0 tx=0).stop 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 shutdown_connections 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c41149c0 0x7f09c411c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f09ac0b6780 0x7f09ac0b8c90 secure :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09c4117130 tx=0x7f09b0005f90 comp rx=0 tx=0).stop 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c410bb90 0x7f09c411be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 --2- 172.21.15.39:0/816945566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f09c410ab30 0x7f09c411b950 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 >> 172.21.15.39:0/816945566 conn(0x7f09c406e9f0 msgr2=0x7f09c410f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 shutdown_connections 2024-09-21T01:27:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:38.220+0000 7f09ca600640 1 -- 172.21.15.39:0/816945566 wait complete. 2024-09-21T01:27:38.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:37 smithi150 ceph-mon[34945]: pgmap v1148: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 363 op/s 2024-09-21T01:27:38.930 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:38 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/816945566' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:27:39.043 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167965 14.67 MB/sec execute 485 sec latency 2992.377 ms 2024-09-21T01:27:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:38 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/816945566' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:27:39.221 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 139.9 --await-for 45.0' 2024-09-21T01:27:39.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 139.9 --await-for 45.0 2024-09-21T01:27:39.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:38 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/816945566' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:27:39.931 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:27:40.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:39 smithi039 ceph-mon[30415]: pgmap v1149: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 392 op/s 2024-09-21T01:27:40.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167965 14.64 MB/sec execute 486 sec latency 3992.426 ms 2024-09-21T01:27:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:39 smithi160 ceph-mon[34696]: pgmap v1149: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 392 op/s 2024-09-21T01:27:40.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:39 smithi150 ceph-mon[34945]: pgmap v1149: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 392 op/s 2024-09-21T01:27:41.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167965 14.61 MB/sec execute 487 sec latency 4992.478 ms 2024-09-21T01:27:42.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167966 14.58 MB/sec execute 488 sec latency 5962.116 ms 2024-09-21T01:27:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:41 smithi039 ceph-mon[30415]: pgmap v1150: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 12 MiB/s wr, 390 op/s 2024-09-21T01:27:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:41 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:41 smithi150 ceph-mon[34945]: pgmap v1150: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 12 MiB/s wr, 390 op/s 2024-09-21T01:27:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:41 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:42.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:41 smithi160 ceph-mon[34696]: pgmap v1150: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 12 MiB/s wr, 390 op/s 2024-09-21T01:27:42.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:41 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- 172.21.15.39:0/207390200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe868103990 msgr2=0x7fe868103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- 172.21.15.39:0/207390200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe868103990 0x7fe868103da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe858009900 tx=0x7fe85802a440 comp rx=0 tx=0).stop 2024-09-21T01:27:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- 172.21.15.39:0/207390200 shutdown_connections 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- 172.21.15.39:0/207390200 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe868109c20 0x7fe868110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- 172.21.15.39:0/207390200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8681092d0 0x7fe8681096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- 172.21.15.39:0/207390200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe868103990 0x7fe868103da0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- 172.21.15.39:0/207390200 >> 172.21.15.39:0/207390200 conn(0x7fe8680ff3e0 msgr2=0x7fe868101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- 172.21.15.39:0/207390200 shutdown_connections 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- 172.21.15.39:0/207390200 wait complete. 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 Processor -- start 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- start start 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 0x7fe86819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8681092d0 0x7fe86819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe868109c20 0x7fe8681a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe868115c10 con 0x7fe8681092d0 2024-09-21T01:27:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe868115a90 con 0x7fe868103990 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.401+0000 7fe867e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe868115d90 con 0x7fe868109c20 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 0x7fe86819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 0x7fe86819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:45484/0 (socket says 172.21.15.39:45484) 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 -- 172.21.15.39:0/767640474 learned_addr learned my addr 172.21.15.39:0/767640474 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866000640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8681092d0 0x7fe86819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 -- 172.21.15.39:0/767640474 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe868109c20 msgr2=0x7fe8681a1e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe868109c20 0x7fe8681a1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 -- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8681092d0 msgr2=0x7fe86819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8681092d0 0x7fe86819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 -- 172.21.15.39:0/767640474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8681a25b0 con 0x7fe868103990 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe866a00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 0x7fe86819c640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe85802aa30 tx=0x7fe858003de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe86ca00640 1 -- 172.21.15.39:0/767640474 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85803f070 con 0x7fe868103990 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe86ca00640 1 -- 172.21.15.39:0/767640474 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe858031d60 con 0x7fe868103990 2024-09-21T01:27:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe86ca00640 1 -- 172.21.15.39:0/767640474 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858035030 con 0x7fe868103990 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8681a2840 con 0x7fe868103990 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.407+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8681a9000 con 0x7fe868103990 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 Processor -- start 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 -- start start 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe86ca00640 1 -- 172.21.15.39:0/767640474 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fe858042070 con 0x7fe868103990 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefef2840 0x55eaf029a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf029aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf029af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55eaf020fca0 con 0x55eaefec9980 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55eaf01ff0d0 con 0x55eaefed1b60 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe874c1d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eaf0202ee0 con 0x55eaefef2840 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf029aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf029aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45488/0 (socket says 172.21.15.39:45488) 2024-09-21T01:27:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 -- 172.21.15.39:0/972222155 learned_addr learned my addr 172.21.15.39:0/972222155 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 -- 172.21.15.39:0/972222155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefef2840 msgr2=0x55eaf029a500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefef2840 0x55eaf029a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 -- 172.21.15.39:0/972222155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 msgr2=0x55eaf029af80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf029af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 -- 172.21.15.39:0/972222155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaf01b1400 con 0x55eaefec9980 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe86ca00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe84c0b67c0 0x7fe84c0b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe866a00640 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf029aa40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe858105550 tx=0x7fe858110580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4f2e03316f89b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe86ca00640 1 -- 172.21.15.39:0/767640474 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7fe858108300 con 0x7fe868103990 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.414+0000 7fe867400640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe84c0b67c0 0x7fe84c0b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe853e00640 1 -- 172.21.15.39:0/972222155 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580d1020 con 0x55eaefec9980 2024-09-21T01:27:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe853e00640 1 -- 172.21.15.39:0/972222155 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8580cd8b0 con 0x55eaefec9980 2024-09-21T01:27:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe853e00640 1 -- 172.21.15.39:0/972222155 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580cdb70 con 0x55eaefec9980 2024-09-21T01:27:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- 172.21.15.39:0/972222155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 msgr2=0x55eaf029aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe867400640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe84c0b67c0 0x7fe84c0b8cd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe86000aab0 tx=0x7fe860009250 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf029aa40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe858105550 tx=0x7fe858110580 comp rx=0 tx=0).stop 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- 172.21.15.39:0/972222155 shutdown_connections 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf029af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf029aa40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- 172.21.15.39:0/972222155 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefef2840 0x55eaf029a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- 172.21.15.39:0/972222155 >> 172.21.15.39:0/972222155 conn(0x55eaefee6990 msgr2=0x55eaf02983f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- 172.21.15.39:0/972222155 shutdown_connections 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- 172.21.15.39:0/972222155 wait complete. 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- start start 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 Processor -- start 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf02a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefed1b60 0x55eaf02a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf02a6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55eaf00607a0 con 0x55eaefec9980 2024-09-21T01:27:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55eaf01a4a10 con 0x55eaefef2840 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55eaf020fca0 con 0x55eaefed1b60 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe874c1d740 10 client.0 _ll_register_callbacks cb 0x55eaf01b4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf02a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf02a5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45500/0 (socket says 172.21.15.39:45500) 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 -- 172.21.15.39:0/295254417 learned_addr learned my addr 172.21.15.39:0/295254417 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 -- 172.21.15.39:0/295254417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefed1b60 msgr2=0x55eaf02a60b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefed1b60 0x55eaf02a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 -- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 msgr2=0x55eaf02a6610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf02a6610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 -- 172.21.15.39:0/295254417 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaeff6cb90 con 0x55eaefec9980 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe866a00640 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf02a5b70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe85802aa00 tx=0x7fe85802a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a51230260602bef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.420+0000 7fe850c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe850c00640 1 -- 172.21.15.39:0/295254417 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580d1070 con 0x55eaefec9980 2024-09-21T01:27:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8580cd8b0 con 0x55eaefec9980 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe850c00640 1 -- 172.21.15.39:0/295254417 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7fe85803b6a0 con 0x55eaefec9980 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe850c00640 1 -- 172.21.15.39:0/295254417 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fe8580cd510 con 0x55eaefec9980 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 10 client.16473 fetch_fsmap learned FSMap version 64 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55eaf02b45b0 con 0x55eaefec9980 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe850c00640 1 -- 172.21.15.39:0/295254417 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8580cd8b0 con 0x55eaefec9980 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe850c00640 1 -- 172.21.15.39:0/295254417 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7fe85807d080 con 0x55eaefec9980 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 10 client.16473 fetch_fsmap finished waiting for FSMap version 64 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 10 client.16473 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 20 client.16473 populate_metadata read hostname 'smithi039' 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaefeb11c0 0x55eaf02b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 4 client.16473 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.425+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eaf0211bf0 con 0x55eaefeb11c0 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.437+0000 7fe866000640 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaefeb11c0 0x55eaf02b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.461+0000 7fe866000640 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaefeb11c0 0x55eaf02b7ab0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adace077aad8c861 server_cookie=8ca70756a10bbe52 in_seq=0 out_seq=0 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.461+0000 7fe850c00640 10 client.16473 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe850c00640 1 -- 172.21.15.39:0/295254417 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55eaf0211bf0 con 0x55eaefeb11c0 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe850c00640 10 client.16473 handle_command_reply: tid=0 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 client.16473 shutdown 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 msgr2=0x55eaf02a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf02a5b70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe85802aa00 tx=0x7fe85802a440 comp rx=0 tx=0).stop 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 shutdown_connections 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaefeb11c0 0x55eaf02b7ab0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf02a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefed1b60 0x55eaf02a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 --2- 172.21.15.39:0/295254417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefec9980 0x55eaf02a5b70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 >> 172.21.15.39:0/295254417 conn(0x55eaefee6990 msgr2=0x55eaf029d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe850c00640 0 client.16473 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 shutdown_connections 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 1 -- 172.21.15.39:0/295254417 wait complete. 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.484+0000 7fe874c1d740 20 client.16473 trim_cache size 0 max 16384 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 Processor -- start 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 -- start start 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefed1b60 0x55eaf02a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf029bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf0297c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe8681b5a20 con 0x55eaefed1b60 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8681b3290 con 0x55eaefef2840 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe874c1d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe8681b1100 con 0x55eaefe86570 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefed1b60 0x55eaf02a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf029bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf029bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:45516/0 (socket says 172.21.15.39:45516) 2024-09-21T01:27:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 -- 172.21.15.39:0/662106119 learned_addr learned my addr 172.21.15.39:0/662106119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe867400640 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf0297c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 -- 172.21.15.39:0/662106119 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 msgr2=0x55eaf0297c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf0297c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 -- 172.21.15.39:0/662106119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefed1b60 msgr2=0x55eaf02a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefed1b60 0x55eaf02a43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 -- 172.21.15.39:0/662106119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaf0298d60 con 0x55eaefef2840 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.519+0000 7fe866000640 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf029bc40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe868108ea0 tx=0x7fe85400bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d124a985375b7676 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe853400640 1 -- 172.21.15.39:0/662106119 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854020020 con 0x55eaefef2840 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe853400640 1 -- 172.21.15.39:0/662106119 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe854014d20 con 0x55eaefef2840 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe853400640 1 -- 172.21.15.39:0/662106119 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854016d10 con 0x55eaefef2840 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/662106119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 msgr2=0x55eaf029bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf029bc40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe868108ea0 tx=0x7fe85400bd10 comp rx=0 tx=0).stop 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/662106119 shutdown_connections 2024-09-21T01:27:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf0297c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefef2840 0x55eaf029bc40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- 172.21.15.39:0/662106119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefed1b60 0x55eaf02a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/662106119 >> 172.21.15.39:0/662106119 conn(0x55eaefee6990 msgr2=0x55eaf029b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/662106119 shutdown_connections 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/662106119 wait complete. 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- start start 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 Processor -- start 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf02a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf02aa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefef2840 0x55eaf02aa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe8681b1750 con 0x55eaefef2840 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8681b2bc0 con 0x55eaefed1b60 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe8681b1e20 con 0x55eaefe86570 2024-09-21T01:27:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 10 client.0 _ll_register_callbacks cb 0x7fe858110340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf02a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf02a9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:33506/0 (socket says 172.21.15.39:33506) 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 -- 172.21.15.39:0/1805316670 learned_addr learned my addr 172.21.15.39:0/1805316670 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866000640 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf02aa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 -- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 msgr2=0x55eaf02aa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf02aa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 -- 172.21.15.39:0/1805316670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefef2840 msgr2=0x55eaf02aa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefef2840 0x55eaf02aa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 -- 172.21.15.39:0/1805316670 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaf0321540 con 0x55eaefe86570 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe866a00640 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf02a9c70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe8580ca370 tx=0x7fe858110580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=277ffc07e8f79e21 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe850c00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe850c00640 1 -- 172.21.15.39:0/1805316670 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580d1070 con 0x55eaefe86570 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe850c00640 1 -- 172.21.15.39:0/1805316670 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7fe85807d080 con 0x55eaefe86570 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe854014d20 con 0x55eaefe86570 2024-09-21T01:27:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe850c00640 1 -- 172.21.15.39:0/1805316670 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fe8580cd6f0 con 0x55eaefe86570 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 10 client.26476 fetch_fsmap learned FSMap version 64 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55eaf0321540 con 0x55eaefe86570 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe850c00640 1 -- 172.21.15.39:0/1805316670 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe854014d20 con 0x55eaefe86570 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe850c00640 1 -- 172.21.15.39:0/1805316670 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7fe858105890 con 0x55eaefe86570 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 10 client.26476 fetch_fsmap finished waiting for FSMap version 64 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 10 client.26476 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 20 client.26476 populate_metadata read hostname 'smithi039' 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaf0322bd0 0x55eaf0324ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 4 client.26476 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.9, "await_for": 45.0}] 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.525+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.9, "await_for": 45.0}) -- 0x55eaf020d6a0 con 0x55eaf0322bd0 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.543+0000 7fe866000640 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaf0322bd0 0x55eaf0324ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.614+0000 7fe866000640 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaf0322bd0 0x55eaf0324ff0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ec2eebba485a4bd server_cookie=5e22cd87053ce52e in_seq=0 out_seq=0 2024-09-21T01:27:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:41.614+0000 7fe850c00640 10 client.26476 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:43.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167966 14.55 MB/sec execute 489 sec latency 6962.163 ms 2024-09-21T01:27:44.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167966 14.52 MB/sec execute 490 sec latency 7962.212 ms 2024-09-21T01:27:44.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:43 smithi039 ceph-mon[30415]: pgmap v1151: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 11 MiB/s wr, 373 op/s 2024-09-21T01:27:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:43 smithi150 ceph-mon[34945]: pgmap v1151: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 11 MiB/s wr, 373 op/s 2024-09-21T01:27:44.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:43 smithi160 ceph-mon[34696]: pgmap v1151: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 11 MiB/s wr, 373 op/s 2024-09-21T01:27:45.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167966 14.49 MB/sec execute 491 sec latency 8962.258 ms 2024-09-21T01:27:46.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167966 14.46 MB/sec execute 492 sec latency 9962.306 ms 2024-09-21T01:27:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:45 smithi150 ceph-mon[34945]: pgmap v1152: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 321 op/s 2024-09-21T01:27:46.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:45 smithi160 ceph-mon[34696]: pgmap v1152: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 321 op/s 2024-09-21T01:27:46.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:45 smithi039 ceph-mon[30415]: pgmap v1152: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 321 op/s 2024-09-21T01:27:47.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167967 14.43 MB/sec execute 493 sec latency 10962.361 ms 2024-09-21T01:27:48.012 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:47 smithi039 ceph-mon[30415]: pgmap v1153: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.6 MiB/s wr, 297 op/s 2024-09-21T01:27:48.012 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:47 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:48.012 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:47 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:48.012 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:47 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:27:48.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167971 14.40 MB/sec execute 494 sec latency 11633.337 ms 2024-09-21T01:27:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:47 smithi150 ceph-mon[34945]: pgmap v1153: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.6 MiB/s wr, 297 op/s 2024-09-21T01:27:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:47 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:47 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:47 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:27:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:47 smithi160 ceph-mon[34696]: pgmap v1153: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.6 MiB/s wr, 297 op/s 2024-09-21T01:27:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:47 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:47 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:47 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:27:49.044 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167971 14.37 MB/sec execute 495 sec latency 12483.355 ms 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.775+0000 7fe850c00640 1 -- 172.21.15.39:0/1805316670 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55eaf020d6a0 con 0x55eaf0322bd0 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.775+0000 7fe850c00640 10 client.26476 handle_command_reply: tid=0 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.775+0000 7fe874c1d740 1 client.26476 shutdown 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.775+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 msgr2=0x55eaf02a9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.775+0000 7fe874c1d740 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf02a9c70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe8580ca370 tx=0x7fe858110580 comp rx=0 tx=0).stop 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 shutdown_connections 2024-09-21T01:27:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55eaf0322bd0 0x55eaf0324ff0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eaefef2840 0x55eaf02aa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe850c00640 0 client.26476 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eaefed1b60 0x55eaf02aa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 --2- 172.21.15.39:0/1805316670 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55eaefe86570 0x55eaf02a9c70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 >> 172.21.15.39:0/1805316670 conn(0x55eaefee6990 msgr2=0x55eaf02b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 shutdown_connections 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.776+0000 7fe874c1d740 1 -- 172.21.15.39:0/1805316670 wait complete. 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.777+0000 7fe874c1d740 20 client.26476 trim_cache size 0 max 16384 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 64, 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:27:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 90, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "82c7c5d8": { 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 90, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.9, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:27:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe84c0b67c0 msgr2=0x7fe84c0b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe84c0b67c0 0x7fe84c0b8cd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe86000aab0 tx=0x7fe860009250 comp rx=0 tx=0).stop 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 msgr2=0x7fe86819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 0x7fe86819c640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe85802aa30 tx=0x7fe858003de0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 shutdown_connections 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe868109c20 0x7fe8681a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8681092d0 0x7fe86819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fe84c0b67c0 0x7fe84c0b8cd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 --2- 172.21.15.39:0/767640474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe868103990 0x7fe86819c640 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 >> 172.21.15.39:0/767640474 conn(0x7fe8680ff3e0 msgr2=0x7fe868101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.778+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 shutdown_connections 2024-09-21T01:27:49.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:49.779+0000 7fe867e00640 1 -- 172.21.15.39:0/767640474 wait complete. 2024-09-21T01:27:50.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.34 MB/sec execute 496 sec latency 12969.652 ms 2024-09-21T01:27:50.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:49 smithi039 ceph-mon[30415]: pgmap v1154: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.9 MiB/s wr, 279 op/s 2024-09-21T01:27:50.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:49 smithi150 ceph-mon[34945]: pgmap v1154: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.9 MiB/s wr, 279 op/s 2024-09-21T01:27:50.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:49 smithi160 ceph-mon[34696]: pgmap v1154: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.9 MiB/s wr, 279 op/s 2024-09-21T01:27:50.638 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '82c7c5d8', quiesce took 16.2 seconds. Will release after: 3.6999999999999993 2024-09-21T01:27:51.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.31 MB/sec execute 497 sec latency 4070.430 ms 2024-09-21T01:27:52.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.29 MB/sec execute 498 sec latency 5070.544 ms 2024-09-21T01:27:52.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:51 smithi150 ceph-mon[34945]: pgmap v1155: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-09-21T01:27:52.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:51 smithi160 ceph-mon[34696]: pgmap v1155: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-09-21T01:27:52.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:51 smithi039 ceph-mon[30415]: pgmap v1155: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-09-21T01:27:53.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.26 MB/sec execute 499 sec latency 6070.599 ms 2024-09-21T01:27:54.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.23 MB/sec execute 500 sec latency 7070.735 ms 2024-09-21T01:27:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:53 smithi150 ceph-mon[34945]: pgmap v1156: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 142 op/s 2024-09-21T01:27:54.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:27:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:53 smithi160 ceph-mon[34696]: pgmap v1156: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 142 op/s 2024-09-21T01:27:54.401 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:53 smithi039 ceph-mon[30415]: pgmap v1156: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 142 op/s 2024-09-21T01:27:54.750 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:27:55.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.20 MB/sec execute 501 sec latency 8070.859 ms 2024-09-21T01:27:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 -- 172.21.15.39:0/1290613643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a7810dc30 msgr2=0x7f7a78110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 --2- 172.21.15.39:0/1290613643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a7810dc30 0x7f7a78110120 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7a70009ca0 tx=0x7f7a7002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 -- 172.21.15.39:0/1290613643 shutdown_connections 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 --2- 172.21.15.39:0/1290613643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a7810dc30 0x7f7a78110120 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 --2- 172.21.15.39:0/1290613643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a78105120 0x7f7a7810d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 --2- 172.21.15.39:0/1290613643 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 0x7f7a78104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 -- 172.21.15.39:0/1290613643 >> 172.21.15.39:0/1290613643 conn(0x7f7a780fe2d0 msgr2=0x7f7a78100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 -- 172.21.15.39:0/1290613643 shutdown_connections 2024-09-21T01:27:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.409+0000 7f7a7f000640 1 -- 172.21.15.39:0/1290613643 wait complete. 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 Processor -- start 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 -- start start 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 0x7f7a7819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a78105120 0x7f7a7819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a7810dc30 0x7f7a781a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7a781138e0 con 0x7f7a78105120 2024-09-21T01:27:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78113760 con 0x7f7a7810dc30 2024-09-21T01:27:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.410+0000 7f7a7f000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78113a60 con 0x7f7a781047d0 2024-09-21T01:27:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a78105120 0x7f7a7819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a78105120 0x7f7a7819c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33398/0 (socket says 172.21.15.39:33398) 2024-09-21T01:27:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7dc00640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 0x7f7a7819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7e600640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a7810dc30 0x7f7a781a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7d200640 1 -- 172.21.15.39:0/3908190967 learned_addr learned my addr 172.21.15.39:0/3908190967 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7dc00640 1 -- 172.21.15.39:0/3908190967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a7810dc30 msgr2=0x7f7a781a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7dc00640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a7810dc30 0x7f7a781a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7dc00640 1 -- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a78105120 msgr2=0x7f7a7819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7dc00640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a78105120 0x7f7a7819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.411+0000 7f7a7dc00640 1 -- 172.21.15.39:0/3908190967 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a781a23c0 con 0x7f7a781047d0 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.412+0000 7f7a7dc00640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 0x7f7a7819c450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00e9f0 tx=0x7f7a6c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.412+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c00ccc0 con 0x7f7a781047d0 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.412+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a6c0108e0 con 0x7f7a781047d0 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.412+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781a26b0 con 0x7f7a781047d0 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.412+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c018880 con 0x7f7a781047d0 2024-09-21T01:27:55.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.412+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a781a8e10 con 0x7f7a781047d0 2024-09-21T01:27:55.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.413+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a78104be0 con 0x7f7a781047d0 2024-09-21T01:27:55.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.416+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7a6c010430 con 0x7f7a781047d0 2024-09-21T01:27:55.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.417+0000 7f7a76000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7a600b6700 0x7f7a600b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.418+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7f7a6c0e82c0 con 0x7f7a781047d0 2024-09-21T01:27:55.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.418+0000 7f7a7d200640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7a600b6700 0x7f7a600b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.418+0000 7f7a7d200640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7a600b6700 0x7f7a600b8c10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7a7810fe00 tx=0x7f7a64005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:55.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.419+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7a6c0b27b0 con 0x7f7a781047d0 2024-09-21T01:27:55.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.658+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a781a90f0 con 0x7f7a781047d0 2024-09-21T01:27:55.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.660+0000 7f7a76000640 1 -- 172.21.15.39:0/3908190967 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11915 (secure 0 0 0) 0x7f7a6c0b76f0 con 0x7f7a781047d0 2024-09-21T01:27:55.661 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:27:55.661 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":64,"btime":"2024-09-21T01:27:24:747931+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:27:24.577857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:27:55.662 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 64 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7a600b6700 msgr2=0x7f7a600b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7a600b6700 0x7f7a600b8c10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7a7810fe00 tx=0x7f7a64005f90 comp rx=0 tx=0).stop 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 msgr2=0x7f7a7819c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 0x7f7a7819c450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00e9f0 tx=0x7f7a6c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 shutdown_connections 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f7a600b6700 0x7f7a600b8c10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a7810dc30 0x7f7a781a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a78105120 0x7f7a7819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 --2- 172.21.15.39:0/3908190967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a781047d0 0x7f7a7819c450 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.662+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 >> 172.21.15.39:0/3908190967 conn(0x7f7a780fe2d0 msgr2=0x7f7a781023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:55.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.663+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 shutdown_connections 2024-09-21T01:27:55.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:55.663+0000 7f7a7f000640 1 -- 172.21.15.39:0/3908190967 wait complete. 2024-09-21T01:27:56.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.17 MB/sec execute 502 sec latency 9070.905 ms 2024-09-21T01:27:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:55 smithi150 ceph-mon[34945]: pgmap v1157: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-09-21T01:27:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:55 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3908190967' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:27:56.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:55 smithi039 ceph-mon[30415]: pgmap v1157: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-09-21T01:27:56.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:55 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3908190967' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:27:56.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:55 smithi160 ceph-mon[34696]: pgmap v1157: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-09-21T01:27:56.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:55 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3908190967' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:27:56.377 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id 82c7c5d8 --release --await' 2024-09-21T01:27:56.377 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id 82c7c5d8 --release --await 2024-09-21T01:27:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:27:57.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 167979 14.14 MB/sec execute 503 sec latency 10070.948 ms 2024-09-21T01:27:57.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 -- 172.21.15.39:0/866927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 msgr2=0x7f288410bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 --2- 172.21.15.39:0/866927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288410bad0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f286c009900 tx=0x7f286c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 -- 172.21.15.39:0/866927242 shutdown_connections 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 --2- 172.21.15.39:0/866927242 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f288410c010 0x7f2884112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 --2- 172.21.15.39:0/866927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288410bad0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 --2- 172.21.15.39:0/866927242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 0x7f2884106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 -- 172.21.15.39:0/866927242 >> 172.21.15.39:0/866927242 conn(0x7f2884101810 msgr2=0x7f2884103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 -- 172.21.15.39:0/866927242 shutdown_connections 2024-09-21T01:27:57.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.421+0000 7f288b200640 1 -- 172.21.15.39:0/866927242 wait complete. 2024-09-21T01:27:57.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.422+0000 7f288b200640 1 Processor -- start 2024-09-21T01:27:57.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 -- start start 2024-09-21T01:27:57.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 0x7f288419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f288410c010 0x7f28841a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2884115d20 con 0x7f288410b6c0 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2884115ba0 con 0x7f2884105d80 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f288b200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2884115ea0 con 0x7f288410c010 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.423+0000 7f287be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 0x7f288419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f287be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288419cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33416/0 (socket says 172.21.15.39:33416) 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f287be00640 1 -- 172.21.15.39:0/3076457237 learned_addr learned my addr 172.21.15.39:0/3076457237 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:57.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2889200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f288410c010 0x7f28841a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 -- 172.21.15.39:0/3076457237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f288410c010 msgr2=0x7f28841a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f288410c010 0x7f28841a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 -- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 msgr2=0x7f288419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 -- 172.21.15.39:0/3076457237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28841a2620 con 0x7f2884105d80 2024-09-21T01:27:57.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f287be00640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:27:57.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.424+0000 7f2883e00640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 0x7f288419c6b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f287400e9f0 tx=0x7f287400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.425+0000 7f2881600640 1 -- 172.21.15.39:0/3076457237 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287400ccc0 con 0x7f2884105d80 2024-09-21T01:27:57.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.425+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28841a2910 con 0x7f2884105d80 2024-09-21T01:27:57.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.425+0000 7f2881600640 1 -- 172.21.15.39:0/3076457237 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28740108e0 con 0x7f2884105d80 2024-09-21T01:27:57.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.425+0000 7f2881600640 1 -- 172.21.15.39:0/3076457237 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874018880 con 0x7f2884105d80 2024-09-21T01:27:57.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.425+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28841a9070 con 0x7f2884105d80 2024-09-21T01:27:57.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.429+0000 7f2891ec3740 1 Processor -- start 2024-09-21T01:27:57.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.429+0000 7f2891ec3740 1 -- start start 2024-09-21T01:27:57.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.429+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c7301c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.429+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c360660 0x55972c730700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.429+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c368840 0x55972c730c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2891ec3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55972c6a7290 con 0x55972c368840 2024-09-21T01:27:57.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2891ec3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55972c6966c0 con 0x55972c360660 2024-09-21T01:27:57.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2891ec3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55972c69a4d0 con 0x55972c389520 2024-09-21T01:27:57.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c7301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c7301c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:41024/0 (socket says 172.21.15.39:41024) 2024-09-21T01:27:57.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 -- 172.21.15.39:0/3964157961 learned_addr learned my addr 172.21.15.39:0/3964157961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f287be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c368840 0x55972c730c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 -- 172.21.15.39:0/3964157961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c360660 msgr2=0x55972c730700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c360660 0x55972c730700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 -- 172.21.15.39:0/3964157961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c368840 msgr2=0x55972c730c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c368840 0x55972c730c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2889200640 1 -- 172.21.15.39:0/3964157961 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55972c623460 con 0x55972c389520 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.430+0000 7f2881600640 1 -- 172.21.15.39:0/3076457237 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2874010430 con 0x7f2884105d80 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.431+0000 7f2889200640 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c7301c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f287c010370 tx=0x7f287c010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b131e413c024ecbb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.431+0000 7f2879600640 1 -- 172.21.15.39:0/3964157961 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c016070 con 0x55972c389520 2024-09-21T01:27:57.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.431+0000 7f2879600640 1 -- 172.21.15.39:0/3964157961 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287c004d20 con 0x55972c389520 2024-09-21T01:27:57.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.431+0000 7f2879600640 1 -- 172.21.15.39:0/3964157961 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c005350 con 0x55972c389520 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 -- 172.21.15.39:0/3964157961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 msgr2=0x55972c7301c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c7301c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f287c010370 tx=0x7f287c010830 comp rx=0 tx=0).stop 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2881600640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f28680b6680 0x7f28680b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 -- 172.21.15.39:0/3964157961 shutdown_connections 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c368840 0x55972c730c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c360660 0x55972c730700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 --2- 172.21.15.39:0/3964157961 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c7301c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 -- 172.21.15.39:0/3964157961 >> 172.21.15.39:0/3964157961 conn(0x55972c37d910 msgr2=0x55972c72e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2881600640 1 -- 172.21.15.39:0/3076457237 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7f28740e7e90 con 0x7f2884105d80 2024-09-21T01:27:57.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.432+0000 7f2891ec3740 1 -- 172.21.15.39:0/3964157961 shutdown_connections 2024-09-21T01:27:57.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.433+0000 7f2889200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f28680b6680 0x7f28680b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.433+0000 7f2891ec3740 1 -- 172.21.15.39:0/3964157961 wait complete. 2024-09-21T01:27:57.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.433+0000 7f2891ec3740 1 -- start start 2024-09-21T01:27:57.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.433+0000 7f2889200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f28680b6680 0x7f28680b8b90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f287c009db0 tx=0x7f287c000f40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 Processor -- start 2024-09-21T01:27:57.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c73be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c73c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c747c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55972c4f7bb0 con 0x55972c360660 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55972c6316c0 con 0x55972c389520 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55972c73e9a0 con 0x55972c368840 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2891ec3740 10 client.0 _ll_register_callbacks cb 0x55972c64c490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f287be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c73c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c73be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f287be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c73c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:41026/0 (socket says 172.21.15.39:41026) 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f287be00640 1 -- 172.21.15.39:0/614496219 learned_addr learned my addr 172.21.15.39:0/614496219 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.434+0000 7f2889200640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c747c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f287be00640 1 -- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 msgr2=0x55972c747c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f287be00640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c747c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f287be00640 1 -- 172.21.15.39:0/614496219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 msgr2=0x55972c73be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f287be00640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c73be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f287be00640 1 -- 172.21.15.39:0/614496219 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55972c3f4b00 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f2883e00640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c73be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f287be00640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c73c3c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f286c000c00 tx=0x7f286c0038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2bb08323f44a890 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.435+0000 7f2872000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2872000640 1 -- 172.21.15.39:0/614496219 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c035040 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2872000640 1 -- 172.21.15.39:0/614496219 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7f286c005020 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f287c004d20 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2872000640 1 -- 172.21.15.39:0/614496219 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f286c003ea0 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2891ec3740 10 client.26494 fetch_fsmap learned FSMap version 64 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55972c46b960 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.436+0000 7f2872000640 1 -- 172.21.15.39:0/614496219 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287c004d20 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2872000640 1 -- 172.21.15.39:0/614496219 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f286c0314a0 con 0x55972c368840 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2891ec3740 10 client.26494 fetch_fsmap finished waiting for FSMap version 64 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2891ec3740 10 client.26494 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:27:57.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2891ec3740 20 client.26494 populate_metadata read hostname 'smithi039' 2024-09-21T01:27:57.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2891ec3740 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c347ea0 0x55972c74dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2891ec3740 4 client.26494 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:27:57.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55972c6a91e0 con 0x55972c347ea0 2024-09-21T01:27:57.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.437+0000 7f2883e00640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c347ea0 0x55972c74dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.438+0000 7f2883e00640 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c347ea0 0x55972c74dcb0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa7ebf06a4664774 server_cookie=c1a874222e224ef6 in_seq=0 out_seq=0 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.438+0000 7f2872000640 10 client.26494 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2872000640 1 -- 172.21.15.39:0/614496219 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55972c6a91e0 con 0x55972c347ea0 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2872000640 10 client.26494 handle_command_reply: tid=0 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 client.26494 shutdown 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 msgr2=0x55972c73c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c73c3c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f286c000c00 tx=0x7f286c0038a0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 shutdown_connections 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c347ea0 0x55972c74dcb0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c747c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c73c3c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 --2- 172.21.15.39:0/614496219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c73be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 >> 172.21.15.39:0/614496219 conn(0x55972c37d910 msgr2=0x55972c733350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2872000640 0 client.26494 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:57.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 shutdown_connections 2024-09-21T01:27:57.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 1 -- 172.21.15.39:0/614496219 wait complete. 2024-09-21T01:27:57.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.439+0000 7f2891ec3740 20 client.26494 trim_cache size 0 max 16384 2024-09-21T01:27:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.458+0000 7f2891ec3740 1 Processor -- start 2024-09-21T01:27:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.458+0000 7f2891ec3740 1 -- start start 2024-09-21T01:27:57.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.458+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c72e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c735420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c735960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2891ec3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f28841b5b60 con 0x55972c360660 2024-09-21T01:27:57.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2891ec3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28841b33d0 con 0x55972c389520 2024-09-21T01:27:57.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2891ec3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f28841b1240 con 0x55972c368840 2024-09-21T01:27:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c72e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c72e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33448/0 (socket says 172.21.15.39:33448) 2024-09-21T01:27:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2883e00640 1 -- 172.21.15.39:0/2124014832 learned_addr learned my addr 172.21.15.39:0/2124014832 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f287be00640 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c735420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f2889200640 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c735960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f287be00640 1 -- 172.21.15.39:0/2124014832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 msgr2=0x55972c735960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f287be00640 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c735960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f287be00640 1 -- 172.21.15.39:0/2124014832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 msgr2=0x55972c72e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f287be00640 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c72e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.459+0000 7f287be00640 1 -- 172.21.15.39:0/2124014832 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55972c73b420 con 0x55972c368840 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.460+0000 7f287be00640 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c735420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f286c005f70 tx=0x7f286c02a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640206e1f21d5228 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.460+0000 7f2878c00640 1 -- 172.21.15.39:0/2124014832 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c035070 con 0x55972c368840 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.460+0000 7f2878c00640 1 -- 172.21.15.39:0/2124014832 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f286c054020 con 0x55972c368840 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.460+0000 7f2878c00640 1 -- 172.21.15.39:0/2124014832 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c031a60 con 0x55972c368840 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 -- 172.21.15.39:0/2124014832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 msgr2=0x55972c735420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c735420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f286c005f70 tx=0x7f286c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 -- 172.21.15.39:0/2124014832 shutdown_connections 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c389520 0x55972c735960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c368840 0x55972c735420 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 --2- 172.21.15.39:0/2124014832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c72e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 -- 172.21.15.39:0/2124014832 >> 172.21.15.39:0/2124014832 conn(0x55972c37d910 msgr2=0x55972c72daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 -- 172.21.15.39:0/2124014832 shutdown_connections 2024-09-21T01:27:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 -- 172.21.15.39:0/2124014832 wait complete. 2024-09-21T01:27:57.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.461+0000 7f2891ec3740 1 -- start start 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 Processor -- start 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c743e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c368840 0x55972c744470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c73ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f28841b1890 con 0x55972c360660 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28841b2d00 con 0x55972c368840 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f28841b1f60 con 0x55972c389520 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.462+0000 7f2891ec3740 10 client.0 _ll_register_callbacks cb 0x7f286c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c73ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c743e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c73ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:41048/0 (socket says 172.21.15.39:41048) 2024-09-21T01:27:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 -- 172.21.15.39:0/4132480428 learned_addr learned my addr 172.21.15.39:0/4132480428 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f287be00640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c368840 0x55972c744470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 -- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c368840 msgr2=0x55972c744470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c368840 0x55972c744470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 -- 172.21.15.39:0/4132480428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 msgr2=0x55972c743e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c743e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 -- 172.21.15.39:0/4132480428 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55972c73f360 con 0x55972c389520 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2883e00640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c743e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2889200640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c73ebd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f287c004ff0 tx=0x7f287c012e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0116e57d0e629 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.463+0000 7f2872000640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2872000640 1 -- 172.21.15.39:0/4132480428 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c039060 con 0x55972c389520 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2872000640 1 -- 172.21.15.39:0/4132480428 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9511+0+0 (secure 0 0 0) 0x7f287c003e00 con 0x55972c389520 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f286c054020 con 0x55972c389520 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2872000640 1 -- 172.21.15.39:0/4132480428 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f287c040050 con 0x55972c389520 2024-09-21T01:27:57.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2891ec3740 10 client.26500 fetch_fsmap learned FSMap version 64 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55972c7b9190 con 0x55972c389520 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.464+0000 7f2872000640 1 -- 172.21.15.39:0/4132480428 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f286c054020 con 0x55972c389520 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2872000640 1 -- 172.21.15.39:0/4132480428 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 64) ==== 8366+0+0 (secure 0 0 0) 0x7f287c033070 con 0x55972c389520 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2891ec3740 10 client.26500 fetch_fsmap finished waiting for FSMap version 64 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2891ec3740 10 client.26500 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2891ec3740 20 client.26500 populate_metadata read hostname 'smithi039' 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2891ec3740 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c31d250 0x55972c7bb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2891ec3740 4 client.26500 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "82c7c5d8", "release": true, "await": true}] 2024-09-21T01:27:57.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "82c7c5d8", "release": true, "await": true}) -- 0x55972c7368c0 con 0x55972c31d250 2024-09-21T01:27:57.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.465+0000 7f2883e00640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c31d250 0x55972c7bb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.466+0000 7f2883e00640 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c31d250 0x55972c7bb660 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb794861211c52b6 server_cookie=9009e481b2ccb139 in_seq=0 out_seq=0 2024-09-21T01:27:57.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.466+0000 7f2872000640 10 client.26500 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:57.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.495+0000 7f2872000640 1 -- 172.21.15.39:0/4132480428 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55972c7368c0 con 0x55972c31d250 2024-09-21T01:27:57.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.495+0000 7f2872000640 10 client.26500 handle_command_reply: tid=0 2024-09-21T01:27:57.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.495+0000 7f2891ec3740 1 client.26500 shutdown 2024-09-21T01:27:57.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 msgr2=0x55972c73ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c73ebd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f287c004ff0 tx=0x7f287c012e00 comp rx=0 tx=0).stop 2024-09-21T01:27:57.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 shutdown_connections 2024-09-21T01:27:57.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55972c31d250 0x55972c7bb660 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55972c389520 0x55972c73ebd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55972c368840 0x55972c744470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 --2- 172.21.15.39:0/4132480428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55972c360660 0x55972c743e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2872000640 0 client.26500 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 >> 172.21.15.39:0/4132480428 conn(0x55972c37d910 msgr2=0x55972c73d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.496+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 shutdown_connections 2024-09-21T01:27:57.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.497+0000 7f2891ec3740 1 -- 172.21.15.39:0/4132480428 wait complete. 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 64, 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 92, 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "82c7c5d8": { 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 92, 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:27:57.499 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.9, 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:27:57.500 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:27:57.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.497+0000 7f2891ec3740 20 client.26500 trim_cache size 0 max 16384 2024-09-21T01:27:57.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f28680b6680 msgr2=0x7f28680b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f28680b6680 0x7f28680b8b90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f287c009db0 tx=0x7f287c000f40 comp rx=0 tx=0).stop 2024-09-21T01:27:57.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 msgr2=0x7f288419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 0x7f288419c6b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f287400e9f0 tx=0x7f287400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 shutdown_connections 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f288410c010 0x7f28841a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288410b6c0 0x7f288419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f28680b6680 0x7f28680b8b90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 --2- 172.21.15.39:0/3076457237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2884105d80 0x7f288419c6b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 >> 172.21.15.39:0/3076457237 conn(0x7f2884101810 msgr2=0x7f2884103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 shutdown_connections 2024-09-21T01:27:57.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:27:57.498+0000 7f288b200640 1 -- 172.21.15.39:0/3076457237 wait complete. 2024-09-21T01:27:58.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 168248 14.15 MB/sec execute 504 sec latency 10514.799 ms 2024-09-21T01:27:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:57 smithi150 ceph-mon[34945]: pgmap v1158: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 487 KiB/s wr, 66 op/s 2024-09-21T01:27:58.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:57 smithi160 ceph-mon[34696]: pgmap v1158: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 487 KiB/s wr, 66 op/s 2024-09-21T01:27:58.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:57 smithi039 ceph-mon[30415]: pgmap v1158: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 487 KiB/s wr, 66 op/s 2024-09-21T01:27:58.522 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '82c7c5d8', total seconds elapsed: 24.1 2024-09-21T01:27:59.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 168764 14.17 MB/sec execute 505 sec latency 105.550 ms 2024-09-21T01:28:00.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 169314 14.19 MB/sec execute 506 sec latency 91.313 ms 2024-09-21T01:28:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:27:59 smithi150 ceph-mon[34945]: pgmap v1159: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 464 KiB/s wr, 55 op/s 2024-09-21T01:28:00.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:27:59 smithi160 ceph-mon[34696]: pgmap v1159: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 464 KiB/s wr, 55 op/s 2024-09-21T01:28:00.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:27:59 smithi039 ceph-mon[30415]: pgmap v1159: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 464 KiB/s wr, 55 op/s 2024-09-21T01:28:01.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 169953 14.25 MB/sec execute 507 sec latency 108.287 ms 2024-09-21T01:28:02.045 INFO:tasks.workunit.client.0.smithi039.stdout: 10 170133 14.24 MB/sec execute 508 sec latency 470.019 ms 2024-09-21T01:28:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:01 smithi150 ceph-mon[34945]: pgmap v1160: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-09-21T01:28:02.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:01 smithi160 ceph-mon[34696]: pgmap v1160: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-09-21T01:28:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:01 smithi039 ceph-mon[30415]: pgmap v1160: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-09-21T01:28:03.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 170254 14.22 MB/sec execute 509 sec latency 647.225 ms 2024-09-21T01:28:04.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 170318 14.19 MB/sec execute 510 sec latency 601.161 ms 2024-09-21T01:28:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:03 smithi150 ceph-mon[34945]: pgmap v1161: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-09-21T01:28:04.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:03 smithi160 ceph-mon[34696]: pgmap v1161: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-09-21T01:28:04.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:03 smithi039 ceph-mon[30415]: pgmap v1161: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-09-21T01:28:05.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 170334 14.17 MB/sec execute 511 sec latency 386.288 ms 2024-09-21T01:28:06.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 170374 14.15 MB/sec execute 512 sec latency 736.987 ms 2024-09-21T01:28:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:05 smithi150 ceph-mon[34945]: pgmap v1162: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 158 op/s 2024-09-21T01:28:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:05 smithi150 ceph-mon[34945]: osdmap e127: 12 total, 12 up, 12 in 2024-09-21T01:28:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:05 smithi160 ceph-mon[34696]: pgmap v1162: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 158 op/s 2024-09-21T01:28:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:05 smithi160 ceph-mon[34696]: osdmap e127: 12 total, 12 up, 12 in 2024-09-21T01:28:06.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:05 smithi039 ceph-mon[30415]: pgmap v1162: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 158 op/s 2024-09-21T01:28:06.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:05 smithi039 ceph-mon[30415]: osdmap e127: 12 total, 12 up, 12 in 2024-09-21T01:28:07.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 170575 14.15 MB/sec execute 513 sec latency 335.279 ms 2024-09-21T01:28:08.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 171060 14.17 MB/sec execute 514 sec latency 650.434 ms 2024-09-21T01:28:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:07 smithi150 ceph-mon[34945]: pgmap v1164: 129 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 119 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-09-21T01:28:08.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:07 smithi160 ceph-mon[34696]: pgmap v1164: 129 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 119 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-09-21T01:28:08.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:07 smithi039 ceph-mon[30415]: pgmap v1164: 129 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 119 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-09-21T01:28:09.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 171302 14.17 MB/sec execute 515 sec latency 1650.487 ms 2024-09-21T01:28:10.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 171769 14.19 MB/sec execute 516 sec latency 2650.544 ms 2024-09-21T01:28:10.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:10 smithi039 ceph-mon[30415]: pgmap v1165: 129 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 119 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.0 MiB/s wr, 472 op/s 2024-09-21T01:28:10.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:10 smithi150 ceph-mon[34945]: pgmap v1165: 129 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 119 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.0 MiB/s wr, 472 op/s 2024-09-21T01:28:10.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:10 smithi160 ceph-mon[34696]: pgmap v1165: 129 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 119 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.0 MiB/s wr, 472 op/s 2024-09-21T01:28:11.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 172324 14.21 MB/sec execute 517 sec latency 3650.594 ms 2024-09-21T01:28:12.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 173022 14.27 MB/sec execute 518 sec latency 4650.636 ms 2024-09-21T01:28:12.302 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:12 smithi039 ceph-mon[30415]: pgmap v1166: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 471 op/s 2024-09-21T01:28:12.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:12 smithi150 ceph-mon[34945]: pgmap v1166: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 471 op/s 2024-09-21T01:28:12.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:12 smithi160 ceph-mon[34696]: pgmap v1166: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 471 op/s 2024-09-21T01:28:13.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 173442 14.30 MB/sec execute 519 sec latency 4989.727 ms 2024-09-21T01:28:14.046 INFO:tasks.workunit.client.0.smithi039.stdout: 10 173803 14.30 MB/sec execute 520 sec latency 249.330 ms 2024-09-21T01:28:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:14 smithi150 ceph-mon[34945]: pgmap v1167: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.2 MiB/s wr, 469 op/s 2024-09-21T01:28:14.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:14 smithi160 ceph-mon[34696]: pgmap v1167: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.2 MiB/s wr, 469 op/s 2024-09-21T01:28:14.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:14 smithi039 ceph-mon[30415]: pgmap v1167: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.2 MiB/s wr, 469 op/s 2024-09-21T01:28:15.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 174138 14.30 MB/sec execute 521 sec latency 229.641 ms 2024-09-21T01:28:16.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 174669 14.32 MB/sec execute 522 sec latency 156.091 ms 2024-09-21T01:28:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:15 smithi150 ceph-mon[34945]: pgmap v1168: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 6.4 MiB/s wr, 495 op/s 2024-09-21T01:28:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:15 smithi150 ceph-mon[34945]: osdmap e128: 12 total, 12 up, 12 in 2024-09-21T01:28:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:15 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:28:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:15 smithi160 ceph-mon[34696]: pgmap v1168: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 6.4 MiB/s wr, 495 op/s 2024-09-21T01:28:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:15 smithi160 ceph-mon[34696]: osdmap e128: 12 total, 12 up, 12 in 2024-09-21T01:28:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:15 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:28:16.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:15 smithi039 ceph-mon[30415]: pgmap v1168: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 6.4 MiB/s wr, 495 op/s 2024-09-21T01:28:16.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:15 smithi039 ceph-mon[30415]: osdmap e128: 12 total, 12 up, 12 in 2024-09-21T01:28:16.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:15 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:28:17.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 175279 14.36 MB/sec execute 523 sec latency 97.558 ms 2024-09-21T01:28:17.824 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:17 smithi039 ceph-mon[30415]: pgmap v1170: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.9 MiB/s wr, 494 op/s 2024-09-21T01:28:18.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 175565 14.36 MB/sec execute 524 sec latency 570.990 ms 2024-09-21T01:28:18.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:17 smithi150 ceph-mon[34945]: pgmap v1170: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.9 MiB/s wr, 494 op/s 2024-09-21T01:28:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:17 smithi160 ceph-mon[34696]: pgmap v1170: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.9 MiB/s wr, 494 op/s 2024-09-21T01:28:19.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 175618 14.34 MB/sec execute 525 sec latency 1571.149 ms 2024-09-21T01:28:20.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 175670 14.32 MB/sec execute 526 sec latency 2571.203 ms 2024-09-21T01:28:20.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:19 smithi150 ceph-mon[34945]: pgmap v1171: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 380 op/s 2024-09-21T01:28:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:19 smithi160 ceph-mon[34696]: pgmap v1171: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 380 op/s 2024-09-21T01:28:20.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:19 smithi039 ceph-mon[30415]: pgmap v1171: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 380 op/s 2024-09-21T01:28:21.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 175773 14.29 MB/sec execute 527 sec latency 3571.256 ms 2024-09-21T01:28:22.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176390 14.33 MB/sec execute 528 sec latency 3592.972 ms 2024-09-21T01:28:22.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:21 smithi150 ceph-mon[34945]: pgmap v1172: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 378 op/s 2024-09-21T01:28:22.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:21 smithi160 ceph-mon[34696]: pgmap v1172: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 378 op/s 2024-09-21T01:28:22.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:21 smithi039 ceph-mon[30415]: pgmap v1172: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 378 op/s 2024-09-21T01:28:23.050 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176802 14.36 MB/sec execute 529 sec latency 452.462 ms 2024-09-21T01:28:24.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:23 smithi039 ceph-mon[30415]: pgmap v1173: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 9.9 MiB/s wr, 386 op/s 2024-09-21T01:28:24.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176804 14.33 MB/sec execute 530 sec latency 1452.527 ms 2024-09-21T01:28:24.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:23 smithi150 ceph-mon[34945]: pgmap v1173: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 9.9 MiB/s wr, 386 op/s 2024-09-21T01:28:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:23 smithi160 ceph-mon[34696]: pgmap v1173: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 9.9 MiB/s wr, 386 op/s 2024-09-21T01:28:25.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176804 14.30 MB/sec execute 531 sec latency 2452.593 ms 2024-09-21T01:28:26.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176810 14.28 MB/sec execute 532 sec latency 3452.657 ms 2024-09-21T01:28:26.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:25 smithi150 ceph-mon[34945]: pgmap v1174: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 370 op/s 2024-09-21T01:28:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:25 smithi160 ceph-mon[34696]: pgmap v1174: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 370 op/s 2024-09-21T01:28:26.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:25 smithi039 ceph-mon[30415]: pgmap v1174: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 370 op/s 2024-09-21T01:28:27.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176815 14.25 MB/sec execute 533 sec latency 4452.726 ms 2024-09-21T01:28:28.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:27 smithi039 ceph-mon[30415]: pgmap v1175: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 10 MiB/s wr, 357 op/s 2024-09-21T01:28:28.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176878 14.23 MB/sec execute 534 sec latency 5452.780 ms 2024-09-21T01:28:28.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:27 smithi150 ceph-mon[34945]: pgmap v1175: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 10 MiB/s wr, 357 op/s 2024-09-21T01:28:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:27 smithi160 ceph-mon[34696]: pgmap v1175: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 10 MiB/s wr, 357 op/s 2024-09-21T01:28:29.047 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176938 14.21 MB/sec execute 535 sec latency 6450.835 ms 2024-09-21T01:28:29.990 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:29 smithi039 ceph-mon[30415]: pgmap v1176: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 11 MiB/s wr, 386 op/s 2024-09-21T01:28:30.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 176995 14.19 MB/sec execute 536 sec latency 7450.898 ms 2024-09-21T01:28:30.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:29 smithi150 ceph-mon[34945]: pgmap v1176: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 11 MiB/s wr, 386 op/s 2024-09-21T01:28:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:29 smithi160 ceph-mon[34696]: pgmap v1176: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 11 MiB/s wr, 386 op/s 2024-09-21T01:28:31.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 177273 14.18 MB/sec execute 537 sec latency 8047.014 ms 2024-09-21T01:28:32.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 177825 14.21 MB/sec execute 538 sec latency 108.563 ms 2024-09-21T01:28:32.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:31 smithi150 ceph-mon[34945]: pgmap v1177: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.2 MiB/s wr, 320 op/s 2024-09-21T01:28:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:31 smithi160 ceph-mon[34696]: pgmap v1177: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.2 MiB/s wr, 320 op/s 2024-09-21T01:28:32.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:31 smithi039 ceph-mon[30415]: pgmap v1177: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.2 MiB/s wr, 320 op/s 2024-09-21T01:28:33.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 178319 14.23 MB/sec execute 539 sec latency 123.162 ms 2024-09-21T01:28:34.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 178851 14.26 MB/sec execute 540 sec latency 106.069 ms 2024-09-21T01:28:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:33 smithi160 ceph-mon[34696]: pgmap v1178: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 285 op/s 2024-09-21T01:28:34.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:33 smithi039 ceph-mon[30415]: pgmap v1178: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 285 op/s 2024-09-21T01:28:34.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:33 smithi150 ceph-mon[34945]: pgmap v1178: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 285 op/s 2024-09-21T01:28:35.048 INFO:tasks.workunit.client.0.smithi039.stdout: 10 179481 14.31 MB/sec execute 541 sec latency 93.383 ms 2024-09-21T01:28:35.480 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T01:28:35.480 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.8 2024-09-21T01:28:35.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:35.904 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:35 smithi039 ceph-mon[30415]: pgmap v1179: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.4 MiB/s wr, 273 op/s 2024-09-21T01:28:36.049 INFO:tasks.workunit.client.0.smithi039.stdout: 10 179944 14.32 MB/sec execute 542 sec latency 123.129 ms 2024-09-21T01:28:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:35 smithi160 ceph-mon[34696]: pgmap v1179: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.4 MiB/s wr, 273 op/s 2024-09-21T01:28:36.313 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:28:36.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:35 smithi150 ceph-mon[34945]: pgmap v1179: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.4 MiB/s wr, 273 op/s 2024-09-21T01:28:37.053 INFO:tasks.workunit.client.0.smithi039.stdout: 10 180285 14.33 MB/sec execute 543 sec latency 360.539 ms 2024-09-21T01:28:38.056 INFO:tasks.workunit.client.0.smithi039.stdout: 10 180305 14.31 MB/sec execute 544 sec latency 1260.481 ms 2024-09-21T01:28:38.127 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:37 smithi039 ceph-mon[30415]: pgmap v1180: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.7 MiB/s wr, 317 op/s 2024-09-21T01:28:38.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:37 smithi150 ceph-mon[34945]: pgmap v1180: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.7 MiB/s wr, 317 op/s 2024-09-21T01:28:38.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:37 smithi160 ceph-mon[34696]: pgmap v1180: 129 pgs: 129 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.7 MiB/s wr, 317 op/s 2024-09-21T01:28:39.057 INFO:tasks.workunit.client.0.smithi039.stdout: 10 180366 14.29 MB/sec execute 545 sec latency 2260.652 ms 2024-09-21T01:28:40.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 180545 14.29 MB/sec execute 546 sec latency 3010.649 ms 2024-09-21T01:28:40.136 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:39 smithi039 ceph-mon[30415]: pgmap v1181: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.9 MiB/s wr, 361 op/s 2024-09-21T01:28:40.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:39 smithi150 ceph-mon[34945]: pgmap v1181: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.9 MiB/s wr, 361 op/s 2024-09-21T01:28:40.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:39 smithi160 ceph-mon[34696]: pgmap v1181: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.9 MiB/s wr, 361 op/s 2024-09-21T01:28:41.057 INFO:tasks.workunit.client.0.smithi039.stdout: 10 180781 14.28 MB/sec execute 547 sec latency 349.275 ms 2024-09-21T01:28:41.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:40 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/821769726' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:40 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/821769726' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":65,"btime":"2024-09-21T01:28:15:010178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":65,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:28:14.581953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 -- 172.21.15.39:0/3584703110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f4105300 msgr2=0x7f39f410d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 --2- 172.21.15.39:0/3584703110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f4105300 0x7f39f410d850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009900 tx=0x7f39e002a410 comp rx=0 tx=0).stop 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 -- 172.21.15.39:0/3584703110 shutdown_connections 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 --2- 172.21.15.39:0/3584703110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39f410de10 0x7f39f4110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 --2- 172.21.15.39:0/3584703110 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f4105300 0x7f39f410d850 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 --2- 172.21.15.39:0/3584703110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f41049b0 0x7f39f4104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 -- 172.21.15.39:0/3584703110 >> 172.21.15.39:0/3584703110 conn(0x7f39f4079730 msgr2=0x7f39f4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 -- 172.21.15.39:0/3584703110 shutdown_connections 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.134+0000 7f39f9800640 1 -- 172.21.15.39:0/3584703110 wait complete. 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.136+0000 7f39f9800640 1 Processor -- start 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.136+0000 7f39f9800640 1 -- start start 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f9800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f41049b0 0x7f39f419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f9800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39f4105300 0x7f39f419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f9800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 0x7f39f41a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f9800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39f4113b20 con 0x7f39f4105300 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f9800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39f41139a0 con 0x7f39f410de10 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f9800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f39f4113ca0 con 0x7f39f41049b0 2024-09-21T01:28:41.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 0x7f39f41a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39f4105300 0x7f39f419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f2a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f41049b0 0x7f39f419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 0x7f39f41a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:48154/0 (socket says 172.21.15.39:48154) 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.138+0000 7f39f3400640 1 -- 172.21.15.39:0/821769726 learned_addr learned my addr 172.21.15.39:0/821769726 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f3400640 1 -- 172.21.15.39:0/821769726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f41049b0 msgr2=0x7f39f419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f3400640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f41049b0 0x7f39f419c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f3400640 1 -- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39f4105300 msgr2=0x7f39f419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f3400640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39f4105300 0x7f39f419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f3400640 1 -- 172.21.15.39:0/821769726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f41a26c0 con 0x7f39f410de10 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f2a00640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f41049b0 0x7f39f419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:41.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.139+0000 7f39f3400640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 0x7f39f41a1f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00eaa0 tx=0x7f39ec00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.140+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec00cdb0 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.140+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39ec005020 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.140+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f41a29b0 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.140+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0116d0 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.140+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f41a9110 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.143+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d40052c0 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.144+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f39ec011870 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.146+0000 7f39eb400640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39d80b6730 0x7f39d80b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.146+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f39ec0e7b40 con 0x7f39f410de10 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.146+0000 7f39f2a00640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39d80b6730 0x7f39d80b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:41.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.147+0000 7f39f2a00640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39d80b6730 0x7f39d80b8c40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f39f406bc60 tx=0x7f39e40090e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.151+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f39ec016030 con 0x7f39f410de10 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.588+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39d4005880 con 0x7f39f410de10 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.592+0000 7f39eb400640 1 -- 172.21.15.39:0/821769726 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+11914 (secure 0 0 0) 0x7f39ec0b2030 con 0x7f39f410de10 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 65 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.598+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39d80b6730 msgr2=0x7f39d80b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39d80b6730 0x7f39d80b8c40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f39f406bc60 tx=0x7f39e40090e0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 msgr2=0x7f39f41a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 0x7f39f41a1f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00eaa0 tx=0x7f39ec00ef60 comp rx=0 tx=0).stop 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 shutdown_connections 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f39d80b6730 0x7f39d80b8c40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39f410de10 0x7f39f41a1f90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39f4105300 0x7f39f419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 --2- 172.21.15.39:0/821769726 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f39f41049b0 0x7f39f419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.599+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 >> 172.21.15.39:0/821769726 conn(0x7f39f4079730 msgr2=0x7f39f41005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:41.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.600+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 shutdown_connections 2024-09-21T01:28:41.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:40.600+0000 7f39f9800640 1 -- 172.21.15.39:0/821769726 wait complete. 2024-09-21T01:28:41.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:40 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/821769726' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:42.042 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:41 smithi039 ceph-mon[30415]: pgmap v1182: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.2 MiB/s wr, 352 op/s 2024-09-21T01:28:42.057 INFO:tasks.workunit.client.0.smithi039.stdout: 10 181270 14.30 MB/sec execute 548 sec latency 154.165 ms 2024-09-21T01:28:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:41 smithi150 ceph-mon[34945]: pgmap v1182: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.2 MiB/s wr, 352 op/s 2024-09-21T01:28:42.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:41 smithi160 ceph-mon[34696]: pgmap v1182: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.2 MiB/s wr, 352 op/s 2024-09-21T01:28:42.365 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0' 2024-09-21T01:28:42.365 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0 2024-09-21T01:28:43.057 INFO:tasks.workunit.client.0.smithi039.stdout: 10 181566 14.29 MB/sec execute 549 sec latency 149.300 ms 2024-09-21T01:28:43.064 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:28:44.057 INFO:tasks.workunit.client.0.smithi039.stdout: 10 181956 14.31 MB/sec execute 550 sec latency 326.039 ms 2024-09-21T01:28:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:43 smithi150 ceph-mon[34945]: pgmap v1183: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 386 op/s 2024-09-21T01:28:44.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:43 smithi160 ceph-mon[34696]: pgmap v1183: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 386 op/s 2024-09-21T01:28:44.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:43 smithi039 ceph-mon[30415]: pgmap v1183: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 386 op/s 2024-09-21T01:28:45.057 INFO:tasks.workunit.client.0.smithi039.stdout: 10 182411 14.33 MB/sec execute 551 sec latency 212.958 ms 2024-09-21T01:28:46.030 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:45 smithi039 ceph-mon[30415]: pgmap v1184: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 405 op/s 2024-09-21T01:28:46.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 182711 14.34 MB/sec execute 552 sec latency 262.363 ms 2024-09-21T01:28:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:45 smithi150 ceph-mon[34945]: pgmap v1184: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 405 op/s 2024-09-21T01:28:46.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:45 smithi160 ceph-mon[34696]: pgmap v1184: 129 pgs: 129 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 405 op/s 2024-09-21T01:28:47.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183106 14.33 MB/sec execute 553 sec latency 253.534 ms 2024-09-21T01:28:47.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 -- 172.21.15.39:0/112560449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c09e590 msgr2=0x7f084c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:47.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 --2- 172.21.15.39:0/112560449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c09e590 0x7f084c0a4dc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0854069e80 tx=0x7f0854098100 comp rx=0 tx=0).stop 2024-09-21T01:28:47.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 -- 172.21.15.39:0/112560449 shutdown_connections 2024-09-21T01:28:47.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 --2- 172.21.15.39:0/112560449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c09e590 0x7f084c0a4dc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 --2- 172.21.15.39:0/112560449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 0x7f084c09e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 --2- 172.21.15.39:0/112560449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c098300 0x7f084c098710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.583+0000 7f0859800640 1 -- 172.21.15.39:0/112560449 >> 172.21.15.39:0/112560449 conn(0x7f084c093d70 msgr2=0x7f084c0961c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:47.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.584+0000 7f0859800640 1 -- 172.21.15.39:0/112560449 shutdown_connections 2024-09-21T01:28:47.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.584+0000 7f0859800640 1 -- 172.21.15.39:0/112560449 wait complete. 2024-09-21T01:28:47.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.585+0000 7f0859800640 1 Processor -- start 2024-09-21T01:28:47.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.586+0000 7f0859800640 1 -- start start 2024-09-21T01:28:47.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.588+0000 7f0859800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 0x7f084c13fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.588+0000 7f0859800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 0x7f084c140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.588+0000 7f0859800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c09e590 0x7f084c1453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.588+0000 7f0853e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 0x7f084c13fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.588+0000 7f0853e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 0x7f084c13fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:44656/0 (socket says 172.21.15.39:44656) 2024-09-21T01:28:47.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.589+0000 7f0859800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f084c0a8170 con 0x7f084c09dc40 2024-09-21T01:28:47.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.589+0000 7f0853e00640 1 -- 172.21.15.39:0/1240080075 learned_addr learned my addr 172.21.15.39:0/1240080075 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:47.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.589+0000 7f0858e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c09e590 0x7f084c1453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.589+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f084c0a7ff0 con 0x7f084c09e590 2024-09-21T01:28:47.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.589+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f084c0a82f0 con 0x7f084c098300 2024-09-21T01:28:47.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.590+0000 7f0853400640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 0x7f084c140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0853e00640 1 -- 172.21.15.39:0/1240080075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c09e590 msgr2=0x7f084c1453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:47.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0853e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c09e590 0x7f084c1453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0853e00640 1 -- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 msgr2=0x7f084c140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:47.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0853e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 0x7f084c140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0853e00640 1 -- 172.21.15.39:0/1240080075 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c145b00 con 0x7f084c098300 2024-09-21T01:28:47.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0858e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c09e590 0x7f084c1453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:28:47.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.591+0000 7f0853400640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 0x7f084c140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:47.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.592+0000 7f0853e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 0x7f084c13fb90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f084800e9f0 tx=0x7f084800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:47.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.592+0000 7f0850c00640 1 -- 172.21.15.39:0/1240080075 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084800cd80 con 0x7f084c098300 2024-09-21T01:28:47.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.593+0000 7f0850c00640 1 -- 172.21.15.39:0/1240080075 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08480108e0 con 0x7f084c098300 2024-09-21T01:28:47.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.593+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c145df0 con 0x7f084c098300 2024-09-21T01:28:47.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.593+0000 7f0850c00640 1 -- 172.21.15.39:0/1240080075 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848018980 con 0x7f084c098300 2024-09-21T01:28:47.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.597+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c14c550 con 0x7f084c098300 2024-09-21T01:28:47.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.605+0000 7f0861be5740 1 Processor -- start 2024-09-21T01:28:47.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.610+0000 7f0850c00640 1 -- 172.21.15.39:0/1240080075 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f0848010430 con 0x7f084c098300 2024-09-21T01:28:47.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.611+0000 7f0861be5740 1 -- start start 2024-09-21T01:28:47.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.613+0000 7f0861be5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c232500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.613+0000 7f0861be5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c232a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.614+0000 7f0861be5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c232f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.614+0000 7f0861be5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558b2c1a7ca0 con 0x558b2be69b60 2024-09-21T01:28:47.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.614+0000 7f0861be5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558b2c1970d0 con 0x558b2be61980 2024-09-21T01:28:47.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.614+0000 7f0861be5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558b2c19aee0 con 0x558b2be8a840 2024-09-21T01:28:47.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.614+0000 7f0858e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c232500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.614+0000 7f0853e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c232a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.615+0000 7f0853e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c232a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:33032/0 (socket says 172.21.15.39:33032) 2024-09-21T01:28:47.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0853e00640 1 -- 172.21.15.39:0/2472347902 learned_addr learned my addr 172.21.15.39:0/2472347902 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:47.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0853400640 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c232f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0858e00640 1 -- 172.21.15.39:0/2472347902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 msgr2=0x558b2c232a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0858e00640 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c232a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0858e00640 1 -- 172.21.15.39:0/2472347902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 msgr2=0x558b2c232f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0858e00640 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c232f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0858e00640 1 -- 172.21.15.39:0/2472347902 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2c149400 con 0x558b2be8a840 2024-09-21T01:28:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.616+0000 7f0853400640 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c232f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:28:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.617+0000 7f0858e00640 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c232500 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0854052860 tx=0x7f0854069e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b80c46d77bd8013 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.617+0000 7f0845600640 1 -- 172.21.15.39:0/2472347902 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854075420 con 0x558b2be8a840 2024-09-21T01:28:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.617+0000 7f0845600640 1 -- 172.21.15.39:0/2472347902 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f085409c040 con 0x558b2be8a840 2024-09-21T01:28:47.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.617+0000 7f0845600640 1 -- 172.21.15.39:0/2472347902 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085406ad30 con 0x558b2be8a840 2024-09-21T01:28:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0861be5740 1 -- 172.21.15.39:0/2472347902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 msgr2=0x558b2c232500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0861be5740 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c232500 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0854052860 tx=0x7f0854069e10 comp rx=0 tx=0).stop 2024-09-21T01:28:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0850c00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f083c0b67d0 0x7f083c0b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0861be5740 1 -- 172.21.15.39:0/2472347902 shutdown_connections 2024-09-21T01:28:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0861be5740 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c232f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0861be5740 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c232a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.620+0000 7f0861be5740 1 --2- 172.21.15.39:0/2472347902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c232500 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.621+0000 7f0861be5740 1 -- 172.21.15.39:0/2472347902 >> 172.21.15.39:0/2472347902 conn(0x558b2be7e990 msgr2=0x558b2c2303f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.621+0000 7f0850c00640 1 -- 172.21.15.39:0/1240080075 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f08480eb170 con 0x7f084c098300 2024-09-21T01:28:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.621+0000 7f0853e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f083c0b67d0 0x7f083c0b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.622+0000 7f0861be5740 1 -- 172.21.15.39:0/2472347902 shutdown_connections 2024-09-21T01:28:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.623+0000 7f0861be5740 1 -- 172.21.15.39:0/2472347902 wait complete. 2024-09-21T01:28:47.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.623+0000 7f0861be5740 1 -- start start 2024-09-21T01:28:47.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.623+0000 7f0853e00640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f083c0b67d0 0x7f083c0b8ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f08480e4f90 tx=0x7f08480e8c50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:47.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 Processor -- start 2024-09-21T01:28:47.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c246ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c247280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c24c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558b2bff87a0 con 0x558b2be69b60 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558b2c13ca10 con 0x558b2be61980 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.624+0000 7f0861be5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558b2c240c80 con 0x558b2be8a840 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.625+0000 7f0853400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c247280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.625+0000 7f0853400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c247280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51022/0 (socket says 172.21.15.39:51022) 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.625+0000 7f0853400640 1 -- 172.21.15.39:0/3297015551 learned_addr learned my addr 172.21.15.39:0/3297015551 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.626+0000 7f0858e00640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c246ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.626+0000 7f0861be5740 10 client.0 _ll_register_callbacks cb 0x558b2c14cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.626+0000 7f0858e00640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c24c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.627+0000 7f0853400640 1 -- 172.21.15.39:0/3297015551 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 msgr2=0x558b2c24c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.627+0000 7f0853400640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c24c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.627+0000 7f0853400640 1 -- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 msgr2=0x558b2c246ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.627+0000 7f0853400640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c246ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.627+0000 7f0853400640 1 -- 172.21.15.39:0/3297015551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2bf04b90 con 0x558b2be69b60 2024-09-21T01:28:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.628+0000 7f0853400640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c247280 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0840005fd0 tx=0x7f0840013ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c942600b246f543 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.628+0000 7f083a000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:28:47.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.629+0000 7f083a000640 1 -- 172.21.15.39:0/3297015551 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840011cc0 con 0x558b2be69b60 2024-09-21T01:28:47.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.629+0000 7f083a000640 1 -- 172.21.15.39:0/3297015551 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f0840004020 con 0x558b2be69b60 2024-09-21T01:28:47.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.629+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f085409c040 con 0x558b2be69b60 2024-09-21T01:28:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.631+0000 7f083a000640 1 -- 172.21.15.39:0/3297015551 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f0840026050 con 0x558b2be69b60 2024-09-21T01:28:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.631+0000 7f0861be5740 10 client.16515 fetch_fsmap learned FSMap version 65 2024-09-21T01:28:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.631+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x558b2c2551d0 con 0x558b2be69b60 2024-09-21T01:28:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.634+0000 7f083a000640 1 -- 172.21.15.39:0/3297015551 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f085409c040 con 0x558b2be69b60 2024-09-21T01:28:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.634+0000 7f083a000640 1 -- 172.21.15.39:0/3297015551 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 65) ==== 8362+0+0 (secure 0 0 0) 0x7f084001b410 con 0x558b2be69b60 2024-09-21T01:28:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.634+0000 7f0861be5740 10 client.16515 fetch_fsmap finished waiting for FSMap version 65 2024-09-21T01:28:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.634+0000 7f0861be5740 10 client.16515 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:28:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.635+0000 7f0861be5740 20 client.16515 populate_metadata read hostname 'smithi039' 2024-09-21T01:28:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.635+0000 7f0861be5740 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be491c0 0x558b2c258780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.635+0000 7f0861be5740 4 client.16515 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:28:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.635+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b2c1a9bf0 con 0x558b2be491c0 2024-09-21T01:28:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.635+0000 7f0858e00640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be491c0 0x558b2c258780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.638+0000 7f083a000640 10 client.16515 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:47.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.638+0000 7f0858e00640 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be491c0 0x558b2c258780 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8345ce58aaf14fa server_cookie=25d52cc685892efc in_seq=0 out_seq=0 2024-09-21T01:28:47.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.640+0000 7f083a000640 1 -- 172.21.15.39:0/3297015551 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558b2c1a9bf0 con 0x558b2be491c0 2024-09-21T01:28:47.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.640+0000 7f083a000640 10 client.16515 handle_command_reply: tid=0 2024-09-21T01:28:47.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.640+0000 7f0861be5740 1 client.16515 shutdown 2024-09-21T01:28:47.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 msgr2=0x558b2c247280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:47.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c247280 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0840005fd0 tx=0x7f0840013ea0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 shutdown_connections 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be491c0 0x558b2c258780 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be8a840 0x558b2c24c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c247280 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 --2- 172.21.15.39:0/3297015551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c246ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f083a000640 0 client.16515 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:47.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 >> 172.21.15.39:0/3297015551 conn(0x558b2be7e990 msgr2=0x558b2c22ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:47.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 shutdown_connections 2024-09-21T01:28:47.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.641+0000 7f0861be5740 1 -- 172.21.15.39:0/3297015551 wait complete. 2024-09-21T01:28:47.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.642+0000 7f0861be5740 20 client.16515 trim_cache size 0 max 16384 2024-09-21T01:28:47.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.688+0000 7f0861be5740 1 Processor -- start 2024-09-21T01:28:47.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.689+0000 7f0861be5740 1 -- start start 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0861be5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be7e990 0x558b2c233530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0861be5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be61980 0x558b2c233a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0861be5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c233fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0861be5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f084c154ce0 con 0x558b2be69b60 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0861be5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f084c152550 con 0x558b2be7e990 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0861be5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f084c150420 con 0x558b2be61980 2024-09-21T01:28:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0858e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be7e990 0x558b2c233530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0858e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c233fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0858e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c233fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51034/0 (socket says 172.21.15.39:51034) 2024-09-21T01:28:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.691+0000 7f0858e00640 1 -- 172.21.15.39:0/1348890616 learned_addr learned my addr 172.21.15.39:0/1348890616 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.692+0000 7f0853400640 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be61980 0x558b2c233a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.692+0000 7f0858e00640 1 -- 172.21.15.39:0/1348890616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be61980 msgr2=0x558b2c233a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.692+0000 7f0858e00640 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be61980 0x558b2c233a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.692+0000 7f0858e00640 1 -- 172.21.15.39:0/1348890616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 msgr2=0x558b2c233fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.692+0000 7f0858e00640 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c233fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.692+0000 7f0858e00640 1 -- 172.21.15.39:0/1348890616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2c236610 con 0x558b2be7e990 2024-09-21T01:28:47.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.693+0000 7f0858e00640 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be7e990 0x558b2c233530 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f08540986f0 tx=0x7f085406e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b288d57120c949fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:47.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.694+0000 7f0844c00640 1 -- 172.21.15.39:0/1348890616 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854098d00 con 0x558b2be7e990 2024-09-21T01:28:47.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.694+0000 7f0844c00640 1 -- 172.21.15.39:0/1348890616 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f085409c070 con 0x558b2be7e990 2024-09-21T01:28:47.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.694+0000 7f0844c00640 1 -- 172.21.15.39:0/1348890616 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08540a17e0 con 0x558b2be7e990 2024-09-21T01:28:47.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 -- 172.21.15.39:0/1348890616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be7e990 msgr2=0x558b2c233530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:47.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be7e990 0x558b2c233530 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f08540986f0 tx=0x7f085406e040 comp rx=0 tx=0).stop 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 -- 172.21.15.39:0/1348890616 shutdown_connections 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c233fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be61980 0x558b2c233a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 --2- 172.21.15.39:0/1348890616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be7e990 0x558b2c233530 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 -- 172.21.15.39:0/1348890616 >> 172.21.15.39:0/1348890616 conn(0x558b2c241490 msgr2=0x558b2c230380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 -- 172.21.15.39:0/1348890616 shutdown_connections 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.695+0000 7f0861be5740 1 -- 172.21.15.39:0/1348890616 wait complete. 2024-09-21T01:28:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.696+0000 7f0861be5740 1 -- start start 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.696+0000 7f0861be5740 1 Processor -- start 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c25e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c25cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be7e990 0x558b2c25d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f084c150a70 con 0x558b2be69b60 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f084c151e80 con 0x558b2be61980 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f084c1510e0 con 0x558b2be7e990 2024-09-21T01:28:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.697+0000 7f0861be5740 10 client.0 _ll_register_callbacks cb 0x7f0840009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:47.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.698+0000 7f0858e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c25e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.698+0000 7f0858e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be7e990 0x558b2c25d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.698+0000 7f0858e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c25e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:33054/0 (socket says 172.21.15.39:33054) 2024-09-21T01:28:47.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.698+0000 7f0858e00640 1 -- 172.21.15.39:0/3716242433 learned_addr learned my addr 172.21.15.39:0/3716242433 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:47.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.698+0000 7f0853400640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c25cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0853400640 1 -- 172.21.15.39:0/3716242433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be7e990 msgr2=0x558b2c25d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0853400640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be7e990 0x558b2c25d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0853400640 1 -- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 msgr2=0x558b2c25e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0853400640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c25e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0853400640 1 -- 172.21.15.39:0/3716242433 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2c25db50 con 0x558b2be69b60 2024-09-21T01:28:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0858e00640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c25e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f0853400640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c25cf70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f084001b9f0 tx=0x7f0840013720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf4792e0eaee5ce server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.699+0000 7f083a000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:28:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.700+0000 7f083a000640 1 -- 172.21.15.39:0/3716242433 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084001bc70 con 0x558b2be69b60 2024-09-21T01:28:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.700+0000 7f083a000640 1 -- 172.21.15.39:0/3716242433 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f0840005040 con 0x558b2be69b60 2024-09-21T01:28:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.700+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f085409c070 con 0x558b2be69b60 2024-09-21T01:28:47.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.701+0000 7f083a000640 1 -- 172.21.15.39:0/3716242433 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f0840005240 con 0x558b2be69b60 2024-09-21T01:28:47.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.701+0000 7f0861be5740 10 client.16521 fetch_fsmap learned FSMap version 65 2024-09-21T01:28:47.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.701+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x558b2c252120 con 0x558b2be69b60 2024-09-21T01:28:47.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.701+0000 7f083a000640 1 -- 172.21.15.39:0/3716242433 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f085409c070 con 0x558b2be69b60 2024-09-21T01:28:47.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.701+0000 7f083a000640 1 -- 172.21.15.39:0/3716242433 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 65) ==== 8362+0+0 (secure 0 0 0) 0x7f0840004c00 con 0x558b2be69b60 2024-09-21T01:28:47.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0861be5740 10 client.16521 fetch_fsmap finished waiting for FSMap version 65 2024-09-21T01:28:47.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0861be5740 10 client.16521 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0861be5740 20 client.16521 populate_metadata read hostname 'smithi039' 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0861be5740 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be8a840 0x558b2c252940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0861be5740 4 client.16521 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}] 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}) -- 0x558b2c1a56a0 con 0x558b2be8a840 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.702+0000 7f0858e00640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be8a840 0x558b2c252940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.703+0000 7f083a000640 10 client.16521 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:47.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:47.703+0000 7f0858e00640 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be8a840 0x558b2c252940 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe5679b3c54a210e server_cookie=7afbaa8a5505f1ee in_seq=0 out_seq=0 2024-09-21T01:28:48.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183349 14.34 MB/sec execute 554 sec latency 455.318 ms 2024-09-21T01:28:48.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.216+0000 7f083a000640 1 -- 172.21.15.39:0/3716242433 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x558b2c1a56a0 con 0x558b2be8a840 2024-09-21T01:28:48.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.216+0000 7f083a000640 10 client.16521 handle_command_reply: tid=0 2024-09-21T01:28:48.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.216+0000 7f0861be5740 1 client.16521 shutdown 2024-09-21T01:28:48.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 msgr2=0x558b2c25cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:48.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c25cf70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f084001b9f0 tx=0x7f0840013720 comp rx=0 tx=0).stop 2024-09-21T01:28:48.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:47 smithi039 ceph-mon[30415]: pgmap v1185: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 418 op/s 2024-09-21T01:28:48.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:47 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 shutdown_connections 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558b2be8a840 0x558b2c252940 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558b2be7e990 0x558b2c25d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b2be69b60 0x558b2c25cf70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 --2- 172.21.15.39:0/3716242433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b2be61980 0x558b2c25e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.217+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 >> 172.21.15.39:0/3716242433 conn(0x558b2c241490 msgr2=0x558b2c230380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.218+0000 7f083a000640 0 client.16521 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.218+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 shutdown_connections 2024-09-21T01:28:48.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.218+0000 7f0861be5740 1 -- 172.21.15.39:0/3716242433 wait complete. 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.219+0000 7f0861be5740 20 client.16521 trim_cache size 0 max 16384 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 65, 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 94, 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:28:48.221 INFO:teuthology.orchestra.run.smithi039.stdout: "e8b32155": { 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 94, 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.8, 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:28:48.222 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.221+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f083c0b67d0 msgr2=0x7f083c0b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:48.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.221+0000 7f0859800640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f083c0b67d0 0x7f083c0b8ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f08480e4f90 tx=0x7f08480e8c50 comp rx=0 tx=0).stop 2024-09-21T01:28:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.221+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 msgr2=0x7f084c13fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.221+0000 7f0859800640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 0x7f084c13fb90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f084800e9f0 tx=0x7f084800eeb0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.222+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 shutdown_connections 2024-09-21T01:28:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.222+0000 7f0859800640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f083c0b67d0 0x7f083c0b8ce0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.222+0000 7f0859800640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c09e590 0x7f084c1453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.222+0000 7f0859800640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f084c09dc40 0x7f084c140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.222+0000 7f0859800640 1 --2- 172.21.15.39:0/1240080075 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f084c098300 0x7f084c13fb90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.222+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 >> 172.21.15.39:0/1240080075 conn(0x7f084c093d70 msgr2=0x7f084c096190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.224+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 shutdown_connections 2024-09-21T01:28:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:48.224+0000 7f0859800640 1 -- 172.21.15.39:0/1240080075 wait complete. 2024-09-21T01:28:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:47 smithi150 ceph-mon[34945]: pgmap v1185: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 418 op/s 2024-09-21T01:28:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:47 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:47 smithi160 ceph-mon[34696]: pgmap v1185: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 418 op/s 2024-09-21T01:28:48.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:47 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:49.046 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e8b32155', quiesce took 13.6 seconds. Will release after: 6.200000000000001 2024-09-21T01:28:49.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.32 MB/sec execute 555 sec latency 1455.418 ms 2024-09-21T01:28:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:28:49.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:28:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:28:50.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.29 MB/sec execute 556 sec latency 2455.493 ms 2024-09-21T01:28:50.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:49 smithi150 ceph-mon[34945]: pgmap v1186: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 13 MiB/s wr, 455 op/s 2024-09-21T01:28:50.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:49 smithi160 ceph-mon[34696]: pgmap v1186: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 13 MiB/s wr, 455 op/s 2024-09-21T01:28:50.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:49 smithi039 ceph-mon[30415]: pgmap v1186: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 13 MiB/s wr, 455 op/s 2024-09-21T01:28:51.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.27 MB/sec execute 557 sec latency 3455.566 ms 2024-09-21T01:28:52.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.24 MB/sec execute 558 sec latency 4455.713 ms 2024-09-21T01:28:52.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:51 smithi150 ceph-mon[34945]: pgmap v1187: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 413 op/s 2024-09-21T01:28:52.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:51 smithi160 ceph-mon[34696]: pgmap v1187: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 413 op/s 2024-09-21T01:28:52.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:51 smithi039 ceph-mon[30415]: pgmap v1187: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 413 op/s 2024-09-21T01:28:53.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.21 MB/sec execute 559 sec latency 5455.786 ms 2024-09-21T01:28:54.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.19 MB/sec execute 560 sec latency 6455.938 ms 2024-09-21T01:28:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:53 smithi150 ceph-mon[34945]: pgmap v1188: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.2 MiB/s wr, 377 op/s 2024-09-21T01:28:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:53 smithi160 ceph-mon[34696]: pgmap v1188: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.2 MiB/s wr, 377 op/s 2024-09-21T01:28:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:53 smithi039 ceph-mon[30415]: pgmap v1188: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.2 MiB/s wr, 377 op/s 2024-09-21T01:28:55.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.16 MB/sec execute 561 sec latency 7456.093 ms 2024-09-21T01:28:55.247 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:55.644 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:28:56.058 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.14 MB/sec execute 562 sec latency 8456.227 ms 2024-09-21T01:28:56.061 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:55 smithi039 ceph-mon[30415]: pgmap v1189: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.4 MiB/s wr, 315 op/s 2024-09-21T01:28:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:55 smithi150 ceph-mon[34945]: pgmap v1189: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.4 MiB/s wr, 315 op/s 2024-09-21T01:28:56.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 -- 172.21.15.39:0/2536195394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810a1b0 msgr2=0x7fad9810a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:56.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 --2- 172.21.15.39:0/2536195394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810a1b0 0x7fad9810a5c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fad88009900 tx=0x7fad8802a410 comp rx=0 tx=0).stop 2024-09-21T01:28:56.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 -- 172.21.15.39:0/2536195394 shutdown_connections 2024-09-21T01:28:56.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 --2- 172.21.15.39:0/2536195394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad98110440 0x7fad98116c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 --2- 172.21.15.39:0/2536195394 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810faf0 0x7fad9810ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 --2- 172.21.15.39:0/2536195394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810a1b0 0x7fad9810a5c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.317+0000 7fad9e400640 1 -- 172.21.15.39:0/2536195394 >> 172.21.15.39:0/2536195394 conn(0x7fad980780e0 msgr2=0x7fad981081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:56.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.318+0000 7fad9e400640 1 -- 172.21.15.39:0/2536195394 shutdown_connections 2024-09-21T01:28:56.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.318+0000 7fad9e400640 1 -- 172.21.15.39:0/2536195394 wait complete. 2024-09-21T01:28:56.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 Processor -- start 2024-09-21T01:28:56.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 -- start start 2024-09-21T01:28:56.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 0x7fad981a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:56.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810faf0 0x7fad981a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:56.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad98110440 0x7fad981a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:56.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fad9811a110 con 0x7fad9810faf0 2024-09-21T01:28:56.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad98119f90 con 0x7fad98110440 2024-09-21T01:28:56.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.319+0000 7fad9e400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fad9811a290 con 0x7fad9810a1b0 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 0x7fad981a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad8ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810faf0 0x7fad981a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 0x7fad981a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:54054/0 (socket says 172.21.15.39:54054) 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 -- 172.21.15.39:0/1304251649 learned_addr learned my addr 172.21.15.39:0/1304251649 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97e00640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad98110440 0x7fad981a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 -- 172.21.15.39:0/1304251649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad98110440 msgr2=0x7fad981a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:56.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad98110440 0x7fad981a63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 -- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810faf0 msgr2=0x7fad981a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810faf0 0x7fad981a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 -- 172.21.15.39:0/1304251649 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad981a6ad0 con 0x7fad9810a1b0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.320+0000 7fad97400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 0x7fad981a0b60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad88005940 tx=0x7fad880040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.321+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8803f030 con 0x7fad9810a1b0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.321+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad981a6d60 con 0x7fad9810a1b0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.321+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad880044c0 con 0x7fad9810a1b0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.321+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad88035070 con 0x7fad9810a1b0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.321+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad981ad520 con 0x7fad9810a1b0 2024-09-21T01:28:56.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.322+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad9810a5c0 con 0x7fad9810a1b0 2024-09-21T01:28:56.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.325+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fad88042030 con 0x7fad9810a1b0 2024-09-21T01:28:56.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.326+0000 7fad94c00640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fad7c0b6680 0x7fad7c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:56.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.326+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7fad88108120 con 0x7fad9810a1b0 2024-09-21T01:28:56.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.326+0000 7fad8ea00640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fad7c0b6680 0x7fad7c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:56.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.327+0000 7fad8ea00640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fad7c0b6680 0x7fad7c0b8b90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fad9806b110 tx=0x7fad80005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:56.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.328+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fad880d2610 con 0x7fad9810a1b0 2024-09-21T01:28:56.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:55 smithi160 ceph-mon[34696]: pgmap v1189: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.4 MiB/s wr, 315 op/s 2024-09-21T01:28:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.565+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad981a1c90 con 0x7fad9810a1b0 2024-09-21T01:28:56.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.566+0000 7fad94c00640 1 -- 172.21.15.39:0/1304251649 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+11914 (secure 0 0 0) 0x7fad880d7550 con 0x7fad9810a1b0 2024-09-21T01:28:56.568 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:28:56.568 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":65,"btime":"2024-09-21T01:28:15:010178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":65,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:28:14.581953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:28:56.568 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 65 2024-09-21T01:28:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fad7c0b6680 msgr2=0x7fad7c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fad7c0b6680 0x7fad7c0b8b90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fad9806b110 tx=0x7fad80005f90 comp rx=0 tx=0).stop 2024-09-21T01:28:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 msgr2=0x7fad981a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 0x7fad981a0b60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad88005940 tx=0x7fad880040c0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 shutdown_connections 2024-09-21T01:28:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fad7c0b6680 0x7fad7c0b8b90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad98110440 0x7fad981a63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9810faf0 0x7fad981a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 --2- 172.21.15.39:0/1304251649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fad9810a1b0 0x7fad981a0b60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 >> 172.21.15.39:0/1304251649 conn(0x7fad980780e0 msgr2=0x7fad981085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 shutdown_connections 2024-09-21T01:28:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:56.569+0000 7fad9e400640 1 -- 172.21.15.39:0/1304251649 wait complete. 2024-09-21T01:28:57.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.11 MB/sec execute 563 sec latency 9456.277 ms 2024-09-21T01:28:57.265 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:56 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1304251649' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:57.303 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id e8b32155 --release --await' 2024-09-21T01:28:57.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id e8b32155 --release --await 2024-09-21T01:28:57.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:56 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1304251649' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:57.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:56 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1304251649' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:57.682 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:28:58.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183351 14.09 MB/sec execute 564 sec latency 10456.331 ms 2024-09-21T01:28:58.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:57 smithi039 ceph-mon[30415]: pgmap v1190: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 6.7 MiB/s wr, 275 op/s 2024-09-21T01:28:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:57 smithi150 ceph-mon[34945]: pgmap v1190: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 6.7 MiB/s wr, 275 op/s 2024-09-21T01:28:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.352+0000 7f13e2600640 1 -- 172.21.15.39:0/359137682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc075750 msgr2=0x7f13dc075b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.352+0000 7f13e2600640 1 --2- 172.21.15.39:0/359137682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc075750 0x7f13dc075b60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009ca0 tx=0x7f13cc02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 -- 172.21.15.39:0/359137682 shutdown_connections 2024-09-21T01:28:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 --2- 172.21.15.39:0/359137682 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc110120 0x7f13dc112540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 --2- 172.21.15.39:0/359137682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc10fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 --2- 172.21.15.39:0/359137682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc075750 0x7f13dc075b60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 -- 172.21.15.39:0/359137682 >> 172.21.15.39:0/359137682 conn(0x7f13dc07b850 msgr2=0x7f13dc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 -- 172.21.15.39:0/359137682 shutdown_connections 2024-09-21T01:28:58.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.353+0000 7f13e2600640 1 -- 172.21.15.39:0/359137682 wait complete. 2024-09-21T01:28:58.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.354+0000 7f13e2600640 1 Processor -- start 2024-09-21T01:28:58.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 -- start start 2024-09-21T01:28:58.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc075750 0x7f13dc19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc110120 0x7f13dc1a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc115c90 con 0x7f13dc110120 2024-09-21T01:28:58.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc115b10 con 0x7f13dc0760a0 2024-09-21T01:28:58.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13e2600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc115e10 con 0x7f13dc075750 2024-09-21T01:28:58.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:57 smithi160 ceph-mon[34696]: pgmap v1190: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 6.7 MiB/s wr, 275 op/s 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13d2a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13db400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc075750 0x7f13dc19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13d2a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:41106/0 (socket says 172.21.15.39:41106) 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13d2a00640 1 -- 172.21.15.39:0/829692025 learned_addr learned my addr 172.21.15.39:0/829692025 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.355+0000 7f13dbe00640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc110120 0x7f13dc1a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d2a00640 1 -- 172.21.15.39:0/829692025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc075750 msgr2=0x7f13dc19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d2a00640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc075750 0x7f13dc19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d2a00640 1 -- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc110120 msgr2=0x7f13dc1a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d2a00640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc110120 0x7f13dc1a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d2a00640 1 -- 172.21.15.39:0/829692025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc1a25f0 con 0x7f13dc0760a0 2024-09-21T01:28:58.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13db400640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc075750 0x7f13dc19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:58.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d2a00640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc19cc20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f13c400e9f0 tx=0x7f13c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13d8c00640 1 -- 172.21.15.39:0/829692025 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c400ccc0 con 0x7f13dc0760a0 2024-09-21T01:28:58.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.356+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13dc1a28e0 con 0x7f13dc0760a0 2024-09-21T01:28:58.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.357+0000 7f13d8c00640 1 -- 172.21.15.39:0/829692025 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13c40108e0 con 0x7f13dc0760a0 2024-09-21T01:28:58.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.357+0000 7f13d8c00640 1 -- 172.21.15.39:0/829692025 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4018860 con 0x7f13dc0760a0 2024-09-21T01:28:58.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.357+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13dc1a9040 con 0x7f13dc0760a0 2024-09-21T01:28:58.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 Processor -- start 2024-09-21T01:28:58.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 -- start start 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae23912520 0x55ae23cb91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 0x55ae23cb9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23cb9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ae23c30290 con 0x55ae238e9660 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55ae23c1f6c0 con 0x55ae23912520 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13e92a1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ae23c234d0 con 0x55ae238f1840 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.360+0000 7f13db400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 0x55ae23cb9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 0x55ae23cb9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49102/0 (socket says 172.21.15.39:49102) 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 -- 172.21.15.39:0/3605755949 learned_addr learned my addr 172.21.15.39:0/3605755949 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:58.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13dbe00640 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae23912520 0x55ae23cb91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 -- 172.21.15.39:0/3605755949 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 msgr2=0x55ae23cb9c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23cb9c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 -- 172.21.15.39:0/3605755949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae23912520 msgr2=0x55ae23cb91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae23912520 0x55ae23cb91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 -- 172.21.15.39:0/3605755949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae23bac460 con 0x55ae238e9660 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13d8c00640 1 -- 172.21.15.39:0/829692025 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f13c4010430 con 0x7f13dc0760a0 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.361+0000 7f13db400640 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 0x55ae23cb9700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f13cc035000 tx=0x7f13cc0094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8652b098a4476ae8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.362+0000 7f13d0c00640 1 -- 172.21.15.39:0/3605755949 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc031650 con 0x55ae238e9660 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.362+0000 7f13d0c00640 1 -- 172.21.15.39:0/3605755949 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13cc036040 con 0x55ae238e9660 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.362+0000 7f13d0c00640 1 -- 172.21.15.39:0/3605755949 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0044a0 con 0x55ae238e9660 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 -- 172.21.15.39:0/3605755949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 msgr2=0x55ae23cb9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 0x55ae23cb9700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f13cc035000 tx=0x7f13cc0094a0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 -- 172.21.15.39:0/3605755949 shutdown_connections 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23cb9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae238e9660 0x55ae23cb9700 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3605755949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae23912520 0x55ae23cb91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 -- 172.21.15.39:0/3605755949 >> 172.21.15.39:0/3605755949 conn(0x55ae23906910 msgr2=0x55ae23cb70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 -- 172.21.15.39:0/3605755949 shutdown_connections 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 -- 172.21.15.39:0/3605755949 wait complete. 2024-09-21T01:28:58.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13d8c00640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13c00b65a0 0x7f13c00b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13d8c00640 1 -- 172.21.15.39:0/829692025 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f13c40e7d80 con 0x7f13dc0760a0 2024-09-21T01:28:58.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13db400640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13c00b65a0 0x7f13c00b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.363+0000 7f13e92a1740 1 -- start start 2024-09-21T01:28:58.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 Processor -- start 2024-09-21T01:28:58.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13db400640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13c00b65a0 0x7f13c00b8ab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0041e0 tx=0x7f13cc005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cc4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cc5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cd0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ae23a80bb0 con 0x55ae23912520 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55ae23bba6c0 con 0x55ae238f1840 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55ae23cc77c0 con 0x55ae238e9660 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.364+0000 7f13e92a1740 10 client.0 _ll_register_callbacks cb 0x55ae23bd5490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cd0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13db400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cc4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cd0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49116/0 (socket says 172.21.15.39:49116) 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13dbe00640 1 -- 172.21.15.39:0/572783210 learned_addr learned my addr 172.21.15.39:0/572783210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13d2a00640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cc5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13d2a00640 1 -- 172.21.15.39:0/572783210 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 msgr2=0x55ae23cc4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13d2a00640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cc4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13d2a00640 1 -- 172.21.15.39:0/572783210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 msgr2=0x55ae23cd0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13d2a00640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cd0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13d2a00640 1 -- 172.21.15.39:0/572783210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae2397db00 con 0x55ae238f1840 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13dbe00640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cd0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.365+0000 7f13db400640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cc4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13d2a00640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cc5330 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f13c40e50f0 tx=0x7f13c4010cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86d483dab8e94281 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13c9600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:28:58.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13c9600640 1 -- 172.21.15.39:0/572783210 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c40ad010 con 0x55ae238f1840 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13cc036040 con 0x55ae238f1840 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13c9600640 1 -- 172.21.15.39:0/572783210 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f13c4093170 con 0x55ae238f1840 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13c9600640 1 -- 172.21.15.39:0/572783210 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f13c4010720 con 0x55ae238f1840 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13e92a1740 10 client.26561 fetch_fsmap learned FSMap version 65 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ae239f4960 con 0x55ae238f1840 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.366+0000 7f13c9600640 1 -- 172.21.15.39:0/572783210 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13cc036040 con 0x55ae238f1840 2024-09-21T01:28:58.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13c9600640 1 -- 172.21.15.39:0/572783210 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 65) ==== 8362+0+0 (secure 0 0 0) 0x7f13c40ac570 con 0x55ae238f1840 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13e92a1740 10 client.26561 fetch_fsmap finished waiting for FSMap version 65 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13e92a1740 10 client.26561 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13e92a1740 20 client.26561 populate_metadata read hostname 'smithi039' 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13e92a1740 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238d0ea0 0x55ae23cd6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13e92a1740 4 client.26561 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae23c321e0 con 0x55ae238d0ea0 2024-09-21T01:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.367+0000 7f13db400640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238d0ea0 0x55ae23cd6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.368+0000 7f13db400640 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238d0ea0 0x55ae23cd6bb0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1019f5f9f21a33 server_cookie=c57c4003aaafa692 in_seq=0 out_seq=0 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.368+0000 7f13c9600640 10 client.26561 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13c9600640 1 -- 172.21.15.39:0/572783210 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ae23c321e0 con 0x55ae238d0ea0 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13c9600640 10 client.26561 handle_command_reply: tid=0 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 client.26561 shutdown 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 msgr2=0x55ae23cc5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cc5330 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f13c40e50f0 tx=0x7f13c4010cc0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 shutdown_connections 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238d0ea0 0x55ae23cd6bb0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cd0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cc5330 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 --2- 172.21.15.39:0/572783210 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cc4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13c9600640 0 client.26561 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 >> 172.21.15.39:0/572783210 conn(0x55ae23906910 msgr2=0x55ae23cbc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 shutdown_connections 2024-09-21T01:28:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 1 -- 172.21.15.39:0/572783210 wait complete. 2024-09-21T01:28:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.369+0000 7f13e92a1740 20 client.26561 trim_cache size 0 max 16384 2024-09-21T01:28:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 Processor -- start 2024-09-21T01:28:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 -- start start 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cb6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cd09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cbe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1b1660 con 0x55ae23912520 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1aeed0 con 0x55ae238f1840 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13e92a1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1acda0 con 0x55ae238e9660 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.388+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cbe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cb6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cb6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:54088/0 (socket says 172.21.15.39:54088) 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 -- 172.21.15.39:0/3020056616 learned_addr learned my addr 172.21.15.39:0/3020056616 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13d2a00640 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cd09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 -- 172.21.15.39:0/3020056616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 msgr2=0x55ae23cd09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cd09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 -- 172.21.15.39:0/3020056616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 msgr2=0x55ae23cbe440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cbe440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 -- 172.21.15.39:0/3020056616 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae23cb53b0 con 0x55ae238e9660 2024-09-21T01:28:58.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.389+0000 7f13db400640 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cb6c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55ae23cb7e40 tx=0x7f13cc0094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=442e3645db228d1f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13cbe00640 1 -- 172.21.15.39:0/3020056616 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc003dc0 con 0x55ae238e9660 2024-09-21T01:28:58.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13cbe00640 1 -- 172.21.15.39:0/3020056616 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13cc069070 con 0x55ae238e9660 2024-09-21T01:28:58.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13cbe00640 1 -- 172.21.15.39:0/3020056616 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc02fa50 con 0x55ae238e9660 2024-09-21T01:28:58.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 -- 172.21.15.39:0/3020056616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 msgr2=0x55ae23cb6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cb6c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55ae23cb7e40 tx=0x7f13cc0094a0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 -- 172.21.15.39:0/3020056616 shutdown_connections 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cbe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238f1840 0x55ae23cd09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 --2- 172.21.15.39:0/3020056616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238e9660 0x55ae23cb6c40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.390+0000 7f13e92a1740 1 -- 172.21.15.39:0/3020056616 >> 172.21.15.39:0/3020056616 conn(0x55ae23906910 msgr2=0x55ae23cbfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.391+0000 7f13e92a1740 1 -- 172.21.15.39:0/3020056616 shutdown_connections 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.391+0000 7f13e92a1740 1 -- 172.21.15.39:0/3020056616 wait complete. 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.391+0000 7f13e92a1740 1 -- start start 2024-09-21T01:28:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.391+0000 7f13e92a1740 1 Processor -- start 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238e9660 0x55ae23cccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23ccd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cc7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1ad3f0 con 0x55ae23912520 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1ae800 con 0x55ae238e9660 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1ada60 con 0x55ae238f1840 2024-09-21T01:28:58.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13e92a1740 10 client.0 _ll_register_callbacks cb 0x7f13c40a9740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cc7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cc7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49140/0 (socket says 172.21.15.39:49140) 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13dbe00640 1 -- 172.21.15.39:0/623278334 learned_addr learned my addr 172.21.15.39:0/623278334 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13d2a00640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23ccd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.392+0000 7f13db400640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238e9660 0x55ae23cccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13dbe00640 1 -- 172.21.15.39:0/623278334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 msgr2=0x55ae23ccd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13dbe00640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23ccd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13dbe00640 1 -- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238e9660 msgr2=0x55ae23cccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13dbe00640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238e9660 0x55ae23cccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13dbe00640 1 -- 172.21.15.39:0/623278334 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae23cc4410 con 0x55ae23912520 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13db400640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238e9660 0x55ae23cccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13dbe00640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cc7a90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f13d4010c40 tx=0x7f13d400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6910a9c3e942be server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13c9600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-09-21T01:28:58.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13c9600640 1 -- 172.21.15.39:0/623278334 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4023060 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13cc069070 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.393+0000 7f13c9600640 1 -- 172.21.15.39:0/623278334 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7f13d401e680 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13c9600640 1 -- 172.21.15.39:0/623278334 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f13d402a050 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 10 client.16545 fetch_fsmap learned FSMap version 65 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ae23d41f10 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13c9600640 1 -- 172.21.15.39:0/623278334 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13cc069070 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13c9600640 1 -- 172.21.15.39:0/623278334 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 65) ==== 8362+0+0 (secure 0 0 0) 0x7f13d400e460 con 0x55ae23912520 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 10 client.16545 fetch_fsmap finished waiting for FSMap version 65 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 10 client.16545 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 20 client.16545 populate_metadata read hostname 'smithi039' 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238a6250 0x55ae23d446e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 4 client.16545 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e8b32155", "release": true, "await": true}] 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.394+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e8b32155", "release": true, "await": true}) -- 0x55ae23c2dc90 con 0x55ae238a6250 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.395+0000 7f13db400640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238a6250 0x55ae23d446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.396+0000 7f13db400640 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238a6250 0x55ae23d446e0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4ee0d693daff5d7 server_cookie=7b5ce86c33ccff0 in_seq=0 out_seq=0 2024-09-21T01:28:58.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.396+0000 7f13c9600640 10 client.16545 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:58.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.431+0000 7f13c9600640 1 -- 172.21.15.39:0/623278334 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ae23c2dc90 con 0x55ae238a6250 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.431+0000 7f13c9600640 10 client.16545 handle_command_reply: tid=0 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.431+0000 7f13e92a1740 1 client.16545 shutdown 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 msgr2=0x55ae23cc7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cc7a90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f13d4010c40 tx=0x7f13d400f790 comp rx=0 tx=0).stop 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 shutdown_connections 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55ae238a6250 0x55ae23d446e0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ae23912520 0x55ae23cc7a90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55ae238f1840 0x55ae23ccd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 --2- 172.21.15.39:0/623278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ae238e9660 0x55ae23cccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13c9600640 0 client.16545 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:28:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 >> 172.21.15.39:0/623278334 conn(0x55ae23906910 msgr2=0x55ae23cbfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 shutdown_connections 2024-09-21T01:28:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.432+0000 7f13e92a1740 1 -- 172.21.15.39:0/623278334 wait complete. 2024-09-21T01:28:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.433+0000 7f13e92a1740 20 client.16545 trim_cache size 0 max 16384 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 65, 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 96, 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "e8b32155": { 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 96, 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:28:58.435 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.8, 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:28:58.436 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:28:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13c00b65a0 msgr2=0x7f13c00b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13c00b65a0 0x7f13c00b8ab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0041e0 tx=0x7f13cc005fd0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 msgr2=0x7f13dc19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc19cc20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f13c400e9f0 tx=0x7f13c400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 shutdown_connections 2024-09-21T01:28:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc110120 0x7f13dc1a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f13c00b65a0 0x7f13c00b8ab0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13dc0760a0 0x7f13dc19cc20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 --2- 172.21.15.39:0/829692025 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f13dc075750 0x7f13dc19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.434+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 >> 172.21.15.39:0/829692025 conn(0x7f13dc07b850 msgr2=0x7f13dc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.435+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 shutdown_connections 2024-09-21T01:28:58.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:28:58.435+0000 7f13e2600640 1 -- 172.21.15.39:0/829692025 wait complete. 2024-09-21T01:28:59.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 183772 14.10 MB/sec execute 565 sec latency 10845.346 ms 2024-09-21T01:28:59.250 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e8b32155', total seconds elapsed: 23.8 2024-09-21T01:28:59.251 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.8 2024-09-21T01:28:59.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:59.878 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:00.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 184288 14.12 MB/sec execute 566 sec latency 89.324 ms 2024-09-21T01:29:00.237 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:28:59 smithi039 ceph-mon[30415]: pgmap v1191: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.3 MiB/s wr, 217 op/s 2024-09-21T01:29:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:28:59 smithi150 ceph-mon[34945]: pgmap v1191: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.3 MiB/s wr, 217 op/s 2024-09-21T01:29:00.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:28:59 smithi160 ceph-mon[34696]: pgmap v1191: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.3 MiB/s wr, 217 op/s 2024-09-21T01:29:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.695+0000 7fa132e00640 1 -- 172.21.15.39:0/654824357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c075750 msgr2=0x7fa12c075b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.695+0000 7fa132e00640 1 --2- 172.21.15.39:0/654824357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c075750 0x7fa12c075b60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa118009900 tx=0x7fa11802a420 comp rx=0 tx=0).stop 2024-09-21T01:29:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 -- 172.21.15.39:0/654824357 shutdown_connections 2024-09-21T01:29:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 --2- 172.21.15.39:0/654824357 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c110120 0x7fa12c112540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 --2- 172.21.15.39:0/654824357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c0760a0 0x7fa12c10fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 --2- 172.21.15.39:0/654824357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c075750 0x7fa12c075b60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:00.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 -- 172.21.15.39:0/654824357 >> 172.21.15.39:0/654824357 conn(0x7fa12c07b850 msgr2=0x7fa12c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:00.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 -- 172.21.15.39:0/654824357 shutdown_connections 2024-09-21T01:29:00.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.696+0000 7fa132e00640 1 -- 172.21.15.39:0/654824357 wait complete. 2024-09-21T01:29:00.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.697+0000 7fa132e00640 1 Processor -- start 2024-09-21T01:29:00.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.697+0000 7fa132e00640 1 -- start start 2024-09-21T01:29:00.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa132e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c075750 0x7fa12c10e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa132e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 0x7fa12c10e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa132e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c110120 0x7fa12c1ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa132e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c104070 con 0x7fa12c075750 2024-09-21T01:29:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa132e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c103ef0 con 0x7fa12c110120 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa132e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1041f0 con 0x7fa12c0760a0 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.698+0000 7fa12b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 0x7fa12c10e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c075750 0x7fa12c10e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 0x7fa12c10e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:54110/0 (socket says 172.21.15.39:54110) 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 -- 172.21.15.39:0/4215302540 learned_addr learned my addr 172.21.15.39:0/4215302540 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa130e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c110120 0x7fa12c1ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 -- 172.21.15.39:0/4215302540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c110120 msgr2=0x7fa12c1ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c110120 0x7fa12c1ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 -- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c075750 msgr2=0x7fa12c10e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:00.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c075750 0x7fa12c10e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:00.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.699+0000 7fa12b400640 1 -- 172.21.15.39:0/4215302540 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c10f000 con 0x7fa12c0760a0 2024-09-21T01:29:00.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.700+0000 7fa12be00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c075750 0x7fa12c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:29:00.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.700+0000 7fa12b400640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 0x7fa12c10e760 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00c970 tx=0x7fa11c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:00.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.701+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c007bb0 con 0x7fa12c0760a0 2024-09-21T01:29:00.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.701+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c007d50 con 0x7fa12c0760a0 2024-09-21T01:29:00.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.701+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00f450 con 0x7fa12c0760a0 2024-09-21T01:29:00.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.701+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12c1acff0 con 0x7fa12c0760a0 2024-09-21T01:29:00.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.701+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12c1ad390 con 0x7fa12c0760a0 2024-09-21T01:29:00.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.703+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa12c075b60 con 0x7fa12c0760a0 2024-09-21T01:29:00.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.706+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa11c004eb0 con 0x7fa12c0760a0 2024-09-21T01:29:00.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.708+0000 7fa128c00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa1100b6700 0x7fa1100b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:00.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.708+0000 7fa12be00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa1100b6700 0x7fa1100b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:00.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.708+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9511+0+0 (secure 0 0 0) 0x7fa11c0e7b20 con 0x7fa12c0760a0 2024-09-21T01:29:00.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.709+0000 7fa12be00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa1100b6700 0x7fa1100b8c10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa118006010 tx=0x7fa118031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:00.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:00.710+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa11c014030 con 0x7fa12c0760a0 2024-09-21T01:29:01.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 184686 14.13 MB/sec execute 567 sec latency 241.187 ms 2024-09-21T01:29:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.151+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa12c10f520 con 0x7fa12c0760a0 2024-09-21T01:29:01.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.152+0000 7fa128c00640 1 -- 172.21.15.39:0/4215302540 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+11914 (secure 0 0 0) 0x7fa11c0adcb0 con 0x7fa12c0760a0 2024-09-21T01:29:01.154 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:29:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 65 2024-09-21T01:29:01.154 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":65,"btime":"2024-09-21T01:28:15:010178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":65,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:28:14.581953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24742,24760,24764]},"id":1}]} 2024-09-21T01:29:01.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.156+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa1100b6700 msgr2=0x7fa1100b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:01.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.156+0000 7fa132e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa1100b6700 0x7fa1100b8c10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa118006010 tx=0x7fa118031400 comp rx=0 tx=0).stop 2024-09-21T01:29:01.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.156+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 msgr2=0x7fa12c10e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:01.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.156+0000 7fa132e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 0x7fa12c10e760 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00c970 tx=0x7fa11c00ce30 comp rx=0 tx=0).stop 2024-09-21T01:29:01.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 shutdown_connections 2024-09-21T01:29:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fa1100b6700 0x7fa1100b8c10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa12c110120 0x7fa12c1ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa12c0760a0 0x7fa12c10e760 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00c970 tx=0x7fa11c00ce30 comp rx=0 tx=0).stop 2024-09-21T01:29:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 --2- 172.21.15.39:0/4215302540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa12c075750 0x7fa12c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 >> 172.21.15.39:0/4215302540 conn(0x7fa12c07b850 msgr2=0x7fa12c1069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 shutdown_connections 2024-09-21T01:29:01.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:01.157+0000 7fa132e00640 1 -- 172.21.15.39:0/4215302540 wait complete. 2024-09-21T01:29:02.001 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0' 2024-09-21T01:29:02.001 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db / --timeout 90 --expiration 139.8 --await-for 45.0 2024-09-21T01:29:02.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 184861 14.12 MB/sec execute 568 sec latency 421.671 ms 2024-09-21T01:29:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:01 smithi150 ceph-mon[34945]: pgmap v1192: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-09-21T01:29:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:01 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4215302540' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:02.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:01 smithi160 ceph-mon[34696]: pgmap v1192: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-09-21T01:29:02.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:01 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4215302540' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:01 smithi039 ceph-mon[30415]: pgmap v1192: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 115 op/s 2024-09-21T01:29:02.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:01 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4215302540' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:02.732 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:03.059 INFO:tasks.workunit.client.0.smithi039.stdout: 10 184963 14.10 MB/sec execute 569 sec latency 862.295 ms 2024-09-21T01:29:04.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 185294 14.12 MB/sec execute 570 sec latency 1185.380 ms 2024-09-21T01:29:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:04 smithi150 ceph-mon[34945]: pgmap v1193: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-09-21T01:29:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:04 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:04.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:04 smithi160 ceph-mon[34696]: pgmap v1193: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-09-21T01:29:04.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:04 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:04.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:04 smithi039 ceph-mon[30415]: pgmap v1193: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 115 op/s 2024-09-21T01:29:04.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:04 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:05.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 185907 14.16 MB/sec execute 571 sec latency 95.355 ms 2024-09-21T01:29:06.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186073 14.15 MB/sec execute 572 sec latency 380.178 ms 2024-09-21T01:29:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:06 smithi150 ceph-mon[34945]: pgmap v1194: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 172 op/s 2024-09-21T01:29:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:06 smithi150 ceph-mon[34945]: osdmap e129: 12 total, 12 up, 12 in 2024-09-21T01:29:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:06 smithi160 ceph-mon[34696]: pgmap v1194: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 172 op/s 2024-09-21T01:29:06.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:06 smithi160 ceph-mon[34696]: osdmap e129: 12 total, 12 up, 12 in 2024-09-21T01:29:06.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:06 smithi039 ceph-mon[30415]: pgmap v1194: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 172 op/s 2024-09-21T01:29:06.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:06 smithi039 ceph-mon[30415]: osdmap e129: 12 total, 12 up, 12 in 2024-09-21T01:29:07.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186393 14.16 MB/sec execute 573 sec latency 206.120 ms 2024-09-21T01:29:07.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.231+0000 7f5c1a800640 1 -- 172.21.15.39:0/83435144 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 msgr2=0x7f5c1410a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.231+0000 7f5c1a800640 1 --2- 172.21.15.39:0/83435144 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c1410a5a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009900 tx=0x7f5c0402a410 comp rx=0 tx=0).stop 2024-09-21T01:29:07.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.233+0000 7f5c1a800640 1 -- 172.21.15.39:0/83435144 shutdown_connections 2024-09-21T01:29:07.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.233+0000 7f5c1a800640 1 --2- 172.21.15.39:0/83435144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c14110420 0x7f5c14116c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.233+0000 7f5c1a800640 1 --2- 172.21.15.39:0/83435144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1410fad0 0x7f5c1410fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.233+0000 7f5c1a800640 1 --2- 172.21.15.39:0/83435144 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c1410a5a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.233+0000 7f5c1a800640 1 -- 172.21.15.39:0/83435144 >> 172.21.15.39:0/83435144 conn(0x7f5c140781c0 msgr2=0x7f5c141081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.234+0000 7f5c1a800640 1 -- 172.21.15.39:0/83435144 shutdown_connections 2024-09-21T01:29:07.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.234+0000 7f5c1a800640 1 -- 172.21.15.39:0/83435144 wait complete. 2024-09-21T01:29:07.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.237+0000 7f5c1a800640 1 Processor -- start 2024-09-21T01:29:07.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.237+0000 7f5c1a800640 1 -- start start 2024-09-21T01:29:07.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.242+0000 7f5c1a800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c141a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.242+0000 7f5c1a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 0x7f5c141a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.242+0000 7f5c1a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c14110420 0x7f5c141a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c13400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c141a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c13400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c141a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59454/0 (socket says 172.21.15.39:59454) 2024-09-21T01:29:07.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 0x7f5c141a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 0x7f5c141a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39852/0 (socket says 172.21.15.39:39852) 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c13400640 1 -- 172.21.15.39:0/2906186089 learned_addr learned my addr 172.21.15.39:0/2906186089 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1411a120 con 0x7f5c14110420 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14119fa0 con 0x7f5c1410fad0 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1411a2a0 con 0x7f5c1410a190 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.243+0000 7f5c13e00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c14110420 0x7f5c141a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.244+0000 7f5c12a00640 1 -- 172.21.15.39:0/2906186089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 msgr2=0x7f5c141a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.244+0000 7f5c12a00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c141a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.244+0000 7f5c12a00640 1 -- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c14110420 msgr2=0x7f5c141a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.244+0000 7f5c12a00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c14110420 0x7f5c141a63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.244+0000 7f5c12a00640 1 -- 172.21.15.39:0/2906186089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c141a6ae0 con 0x7f5c1410fad0 2024-09-21T01:29:07.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.244+0000 7f5c12a00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 0x7f5c141a1110 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5c1406c940 tx=0x7f5c0000cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:07.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.246+0000 7f5c0be00640 1 -- 172.21.15.39:0/2906186089 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007bb0 con 0x7f5c1410fad0 2024-09-21T01:29:07.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.246+0000 7f5c0be00640 1 -- 172.21.15.39:0/2906186089 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c00007d50 con 0x7f5c1410fad0 2024-09-21T01:29:07.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.246+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c141a6dd0 con 0x7f5c1410fad0 2024-09-21T01:29:07.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.246+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c141ad530 con 0x7f5c1410fad0 2024-09-21T01:29:07.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.246+0000 7f5c0be00640 1 -- 172.21.15.39:0/2906186089 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00004de0 con 0x7f5c1410fad0 2024-09-21T01:29:07.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.252+0000 7f5c214b4740 1 Processor -- start 2024-09-21T01:29:07.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.252+0000 7f5c214b4740 1 -- start start 2024-09-21T01:29:07.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.253+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6d1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.254+0000 7f5c0be00640 1 -- 172.21.15.39:0/2906186089 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5c0000f450 con 0x7f5c1410fad0 2024-09-21T01:29:07.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.254+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6d1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.254+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.254+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6d1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59458/0 (socket says 172.21.15.39:59458) 2024-09-21T01:29:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.254+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.255+0000 7f5c214b4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f53a646ca0 con 0x55f53a300980 2024-09-21T01:29:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.255+0000 7f5c214b4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55f53a6360d0 con 0x55f53a308b60 2024-09-21T01:29:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.255+0000 7f5c214b4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f53a639ee0 con 0x55f53a329840 2024-09-21T01:29:07.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.255+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.255+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39856/0 (socket says 172.21.15.39:39856) 2024-09-21T01:29:07.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.255+0000 7f5c12a00640 1 -- 172.21.15.39:0/416768135 learned_addr learned my addr 172.21.15.39:0/416768135 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:07.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.256+0000 7f5c13400640 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6d1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.257+0000 7f5c12a00640 1 -- 172.21.15.39:0/416768135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 msgr2=0x55f53a6d1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.257+0000 7f5c12a00640 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6d1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.257+0000 7f5c12a00640 1 -- 172.21.15.39:0/416768135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 msgr2=0x55f53a6d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.257+0000 7f5c12a00640 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6d1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.257+0000 7f5c12a00640 1 -- 172.21.15.39:0/416768135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f53a5e8400 con 0x55f53a308b60 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.257+0000 7f5c0be00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bf80b67d0 0x7f5bf80b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.258+0000 7f5c13400640 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6d1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:29:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.258+0000 7f5c0be00640 1 -- 172.21.15.39:0/2906186089 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9599+0+0 (secure 0 0 0) 0x7f5c000e78c0 con 0x7f5c1410fad0 2024-09-21T01:29:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.259+0000 7f5c13e00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bf80b67d0 0x7f5bf80b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.260+0000 7f5c12a00640 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6d1dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c000e4ac0 tx=0x7f5c000edcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efa3bbd88e77de07 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:07.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.262+0000 7f5c08c00640 1 -- 172.21.15.39:0/416768135 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0001d030 con 0x55f53a308b60 2024-09-21T01:29:07.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.262+0000 7f5c13e00640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bf80b67d0 0x7f5bf80b8ce0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c005fd0 tx=0x7f5c0c009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:07.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.262+0000 7f5c08c00640 1 -- 172.21.15.39:0/416768135 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c000ede40 con 0x55f53a308b60 2024-09-21T01:29:07.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.264+0000 7f5c08c00640 1 -- 172.21.15.39:0/416768135 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c000ad050 con 0x55f53a308b60 2024-09-21T01:29:07.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.264+0000 7f5c214b4740 1 -- 172.21.15.39:0/416768135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 msgr2=0x55f53a6d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.264+0000 7f5c214b4740 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6d1dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c000e4ac0 tx=0x7f5c000edcb0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.266+0000 7f5c214b4740 1 -- 172.21.15.39:0/416768135 shutdown_connections 2024-09-21T01:29:07.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.266+0000 7f5c214b4740 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6d1dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.266+0000 7f5c214b4740 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6d1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.266+0000 7f5c214b4740 1 --2- 172.21.15.39:0/416768135 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.266+0000 7f5c214b4740 1 -- 172.21.15.39:0/416768135 >> 172.21.15.39:0/416768135 conn(0x55f53a31d990 msgr2=0x55f53a6cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.266+0000 7f5c214b4740 1 -- 172.21.15.39:0/416768135 shutdown_connections 2024-09-21T01:29:07.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.267+0000 7f5c214b4740 1 -- 172.21.15.39:0/416768135 wait complete. 2024-09-21T01:29:07.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.268+0000 7f5c214b4740 1 -- start start 2024-09-21T01:29:07.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.268+0000 7f5c214b4740 1 Processor -- start 2024-09-21T01:29:07.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.270+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6dc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.270+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6dcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.270+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6dd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.271+0000 7f5c13400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6dc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.272+0000 7f5c13400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6dc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60384/0 (socket says 172.21.15.39:60384) 2024-09-21T01:29:07.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.272+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6dcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6dcee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39864/0 (socket says 172.21.15.39:39864) 2024-09-21T01:29:07.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c214b4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f53a4977a0 con 0x55f53a300980 2024-09-21T01:29:07.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c13400640 1 -- 172.21.15.39:0/1418374605 learned_addr learned my addr 172.21.15.39:0/1418374605 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:07.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c214b4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55f53a5dba10 con 0x55f53a308b60 2024-09-21T01:29:07.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55f53a6dfbd0 con 0x55f53a329840 2024-09-21T01:29:07.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c214b4740 10 client.0 _ll_register_callbacks cb 0x55f53a5ebbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:29:07.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.273+0000 7f5c13e00640 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6dd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.275+0000 7f5c13e00640 1 -- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 msgr2=0x55f53a6dcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.275+0000 7f5c13e00640 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6dcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.275+0000 7f5c13e00640 1 -- 172.21.15.39:0/1418374605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 msgr2=0x55f53a6dc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.275+0000 7f5c13e00640 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6dc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.275+0000 7f5c13e00640 1 -- 172.21.15.39:0/1418374605 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f53a3a3b90 con 0x55f53a329840 2024-09-21T01:29:07.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.276+0000 7f5c13e00640 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6dd420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c011c40 tx=0x7f5c0c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113a423e4abd74e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:07.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.276+0000 7f5bfd600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:29:07.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.276+0000 7f5bfd600640 1 -- 172.21.15.39:0/1418374605 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00fb50 con 0x55f53a329840 2024-09-21T01:29:07.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.276+0000 7f5bfd600640 1 -- 172.21.15.39:0/1418374605 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9599+0+0 (secure 0 0 0) 0x7f5c0c00fcf0 con 0x55f53a329840 2024-09-21T01:29:07.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.280+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f53a648bf0 con 0x55f53a329840 2024-09-21T01:29:07.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.280+0000 7f5bfd600640 1 -- 172.21.15.39:0/1418374605 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f5c0c01f420 con 0x55f53a329840 2024-09-21T01:29:07.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.281+0000 7f5c214b4740 10 client.26560 fetch_fsmap learned FSMap version 66 2024-09-21T01:29:07.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.281+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55f53a6eb900 con 0x55f53a329840 2024-09-21T01:29:07.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.281+0000 7f5bfd600640 1 -- 172.21.15.39:0/1418374605 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f53a648bf0 con 0x55f53a329840 2024-09-21T01:29:07.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.281+0000 7f5bfd600640 1 -- 172.21.15.39:0/1418374605 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 66) ==== 8358+0+0 (secure 0 0 0) 0x7f5c0c031660 con 0x55f53a329840 2024-09-21T01:29:07.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.282+0000 7f5c214b4740 10 client.26560 fetch_fsmap finished waiting for FSMap version 66 2024-09-21T01:29:07.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.282+0000 7f5c214b4740 10 client.26560 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:29:07.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.282+0000 7f5c214b4740 20 client.26560 populate_metadata read hostname 'smithi039' 2024-09-21T01:29:07.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.283+0000 7f5c214b4740 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2e81c0 0x55f53a6eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.283+0000 7f5c214b4740 4 client.26560 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:29:07.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.283+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f53a6446a0 con 0x55f53a2e81c0 2024-09-21T01:29:07.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.285+0000 7f5c13400640 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2e81c0 0x55f53a6eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.287+0000 7f5c13400640 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2e81c0 0x55f53a6eeeb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37847919d6440bc3 server_cookie=38332fa970304071 in_seq=0 out_seq=0 2024-09-21T01:29:07.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.287+0000 7f5bfd600640 10 client.26560 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:07.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.288+0000 7f5bfd600640 1 -- 172.21.15.39:0/1418374605 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f53a6446a0 con 0x55f53a2e81c0 2024-09-21T01:29:07.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.288+0000 7f5bfd600640 10 client.26560 handle_command_reply: tid=0 2024-09-21T01:29:07.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.288+0000 7f5c214b4740 1 client.26560 shutdown 2024-09-21T01:29:07.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.289+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 msgr2=0x55f53a6dd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.289+0000 7f5c214b4740 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6dd420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c011c40 tx=0x7f5c0c004a40 comp rx=0 tx=0).stop 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 shutdown_connections 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2e81c0 0x55f53a6eeeb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a329840 0x55f53a6dd420 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6dcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 --2- 172.21.15.39:0/1418374605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a300980 0x55f53a6dc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 >> 172.21.15.39:0/1418374605 conn(0x55f53a31d990 msgr2=0x55f53a6d44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5bfd600640 0 client.26560 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 shutdown_connections 2024-09-21T01:29:07.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.290+0000 7f5c214b4740 1 -- 172.21.15.39:0/1418374605 wait complete. 2024-09-21T01:29:07.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.291+0000 7f5c214b4740 20 client.26560 trim_cache size 0 max 16384 2024-09-21T01:29:07.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 Processor -- start 2024-09-21T01:29:07.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 -- start start 2024-09-21T01:29:07.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6de880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a308b60 0x55f53a6ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 0x55f53a6dcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf4012a40 con 0x55f53a308b60 2024-09-21T01:29:07.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf40102e0 con 0x55f53a329840 2024-09-21T01:29:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.327+0000 7f5c214b4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf400e150 con 0x55f53a300980 2024-09-21T01:29:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.328+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 0x55f53a6dcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.328+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 0x55f53a6dcea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39872/0 (socket says 172.21.15.39:39872) 2024-09-21T01:29:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.328+0000 7f5c13e00640 1 -- 172.21.15.39:0/150749393 learned_addr learned my addr 172.21.15.39:0/150749393 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5c13e00640 1 -- 172.21.15.39:0/150749393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 msgr2=0x55f53a6de880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5c13e00640 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6de880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5c13e00640 1 -- 172.21.15.39:0/150749393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a308b60 msgr2=0x55f53a6ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5c13e00640 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a308b60 0x55f53a6ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5c13e00640 1 -- 172.21.15.39:0/150749393 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f53a6cac50 con 0x55f53a329840 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5c13e00640 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 0x55f53a6dcea0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c031b50 tx=0x7f5c0c005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c2ee9922717e799 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.329+0000 7f5bffe00640 1 -- 172.21.15.39:0/150749393 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c031620 con 0x55f53a329840 2024-09-21T01:29:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.330+0000 7f5bffe00640 1 -- 172.21.15.39:0/150749393 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c0c0317c0 con 0x55f53a329840 2024-09-21T01:29:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.330+0000 7f5bffe00640 1 -- 172.21.15.39:0/150749393 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c01fc00 con 0x55f53a329840 2024-09-21T01:29:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 -- 172.21.15.39:0/150749393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 msgr2=0x55f53a6dcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 0x55f53a6dcea0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c031b50 tx=0x7f5c0c005c80 comp rx=0 tx=0).stop 2024-09-21T01:29:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 -- 172.21.15.39:0/150749393 shutdown_connections 2024-09-21T01:29:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a329840 0x55f53a6dcea0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a308b60 0x55f53a6ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 --2- 172.21.15.39:0/150749393 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 -- 172.21.15.39:0/150749393 >> 172.21.15.39:0/150749393 conn(0x55f53a31d990 msgr2=0x55f53a6cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 -- 172.21.15.39:0/150749393 shutdown_connections 2024-09-21T01:29:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.331+0000 7f5c214b4740 1 -- 172.21.15.39:0/150749393 wait complete. 2024-09-21T01:29:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.332+0000 7f5c214b4740 1 -- start start 2024-09-21T01:29:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.332+0000 7f5c214b4740 1 Processor -- start 2024-09-21T01:29:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.334+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.334+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.334+0000 7f5c214b4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a329840 0x55f53a6e0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.334+0000 7f5c13400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.334+0000 7f5c13400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6e55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:59482/0 (socket says 172.21.15.39:59482) 2024-09-21T01:29:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c214b4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf400e7a0 con 0x55f53a329840 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c214b4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf400fc10 con 0x55f53a308b60 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c214b4740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf400ee70 con 0x55f53a300980 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c214b4740 10 client.0 _ll_register_callbacks cb 0x7f5c0c009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6e0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39882/0 (socket says 172.21.15.39:39882) 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.335+0000 7f5c12a00640 1 -- 172.21.15.39:0/2437047680 learned_addr learned my addr 172.21.15.39:0/2437047680 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.336+0000 7f5c13e00640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a329840 0x55f53a6e0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.337+0000 7f5c13400640 1 -- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 msgr2=0x55f53a6e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.337+0000 7f5c13400640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.337+0000 7f5c13400640 1 -- 172.21.15.39:0/2437047680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a329840 msgr2=0x55f53a6e0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:07.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.337+0000 7f5c13400640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a329840 0x55f53a6e0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.337+0000 7f5c13400640 1 -- 172.21.15.39:0/2437047680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f53a6e1630 con 0x55f53a300980 2024-09-21T01:29:07.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.338+0000 7f5c13e00640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a329840 0x55f53a6e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:29:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.339+0000 7f5c13400640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6e55d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55f53a6ef620 tx=0x7f5c0402a9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a89747be25872e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.339+0000 7f5bfd600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:29:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.344+0000 7f5bfd600640 1 -- 172.21.15.39:0/2437047680 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c040049e0 con 0x55f53a300980 2024-09-21T01:29:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.344+0000 7f5bfd600640 1 -- 172.21.15.39:0/2437047680 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9599+0+0 (secure 0 0 0) 0x7f5c04031850 con 0x55f53a300980 2024-09-21T01:29:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.344+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c0c0317c0 con 0x55f53a300980 2024-09-21T01:29:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.348+0000 7f5bfd600640 1 -- 172.21.15.39:0/2437047680 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f5c04031a50 con 0x55f53a300980 2024-09-21T01:29:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.348+0000 7f5c214b4740 10 client.26563 fetch_fsmap learned FSMap version 66 2024-09-21T01:29:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.348+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55f53a7596f0 con 0x55f53a300980 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.350+0000 7f5bfd600640 1 -- 172.21.15.39:0/2437047680 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c0c0317c0 con 0x55f53a300980 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.350+0000 7f5bfd600640 1 -- 172.21.15.39:0/2437047680 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 66) ==== 8358+0+0 (secure 0 0 0) 0x7f5c0405a420 con 0x55f53a300980 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c214b4740 10 client.26563 fetch_fsmap finished waiting for FSMap version 66 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c214b4740 10 client.26563 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c214b4740 20 client.26563 populate_metadata read hostname 'smithi039' 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c214b4740 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2bd570 0x55f53a75bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c214b4740 4 client.26563 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}] 2024-09-21T01:29:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.8, "await_for": 45.0}) -- 0x55f53a6d8550 con 0x55f53a2bd570 2024-09-21T01:29:07.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.351+0000 7f5c12a00640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2bd570 0x55f53a75bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:07.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.355+0000 7f5c12a00640 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2bd570 0x55f53a75bf20 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fbe578b53d37c1a server_cookie=34c27e5746e00536 in_seq=0 out_seq=0 2024-09-21T01:29:07.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.355+0000 7f5bfd600640 10 client.26563 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.970+0000 7f5bfd600640 1 -- 172.21.15.39:0/2437047680 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f53a6d8550 con 0x55f53a2bd570 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.970+0000 7f5bfd600640 10 client.26563 handle_command_reply: tid=0 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.970+0000 7f5c214b4740 1 client.26563 shutdown 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.970+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 msgr2=0x55f53a6e55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.970+0000 7f5c214b4740 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6e55d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55f53a6ef620 tx=0x7f5c0402a9b0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 shutdown_connections 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55f53a2bd570 0x55f53a75bf20 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f53a329840 0x55f53a6e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f53a308b60 0x55f53a6e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 --2- 172.21.15.39:0/2437047680 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55f53a300980 0x55f53a6e55d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 >> 172.21.15.39:0/2437047680 conn(0x55f53a31d990 msgr2=0x55f53a6de5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5bfd600640 0 client.26563 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 shutdown_connections 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.971+0000 7f5c214b4740 1 -- 172.21.15.39:0/2437047680 wait complete. 2024-09-21T01:29:07.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.972+0000 7f5c214b4740 20 client.26563 trim_cache size 0 max 16384 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 66, 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 98, 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "ff810f0f": { 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 98, 2024-09-21T01:29:07.974 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.8, 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "QUIESCED", 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:07.975 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.974+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bf80b67d0 msgr2=0x7f5bf80b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.974+0000 7f5c1a800640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bf80b67d0 0x7f5bf80b8ce0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c005fd0 tx=0x7f5c0c009500 comp rx=0 tx=0).stop 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.974+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 msgr2=0x7f5c141a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.974+0000 7f5c1a800640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 0x7f5c141a1110 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5c1406c940 tx=0x7f5c0000cdd0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 shutdown_connections 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c14110420 0x7f5c141a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f5bf80b67d0 0x7f5bf80b8ce0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1410fad0 0x7f5c141a1110 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 --2- 172.21.15.39:0/2906186089 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5c1410a190 0x7f5c141a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 >> 172.21.15.39:0/2906186089 conn(0x7f5c140781c0 msgr2=0x7f5c141087c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 shutdown_connections 2024-09-21T01:29:07.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:07.978+0000 7f5c1a800640 1 -- 172.21.15.39:0/2906186089 wait complete. 2024-09-21T01:29:08.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.14 MB/sec execute 574 sec latency 558.554 ms 2024-09-21T01:29:08.310 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:08 smithi039 ceph-mon[30415]: pgmap v1196: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.6 MiB/s wr, 247 op/s 2024-09-21T01:29:08.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:08 smithi150 ceph-mon[34945]: pgmap v1196: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.6 MiB/s wr, 247 op/s 2024-09-21T01:29:08.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:08 smithi160 ceph-mon[34696]: pgmap v1196: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.6 MiB/s wr, 247 op/s 2024-09-21T01:29:08.926 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ff810f0f', quiesce took 9.7 seconds. Will release after: 10.100000000000001 2024-09-21T01:29:09.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.12 MB/sec execute 575 sec latency 1497.933 ms 2024-09-21T01:29:10.060 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.09 MB/sec execute 576 sec latency 2498.094 ms 2024-09-21T01:29:10.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:10 smithi150 ceph-mon[34945]: pgmap v1197: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 406 op/s 2024-09-21T01:29:10.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:10 smithi160 ceph-mon[34696]: pgmap v1197: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 406 op/s 2024-09-21T01:29:10.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:10 smithi039 ceph-mon[30415]: pgmap v1197: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 406 op/s 2024-09-21T01:29:11.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.07 MB/sec execute 577 sec latency 3498.242 ms 2024-09-21T01:29:12.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.05 MB/sec execute 578 sec latency 4498.317 ms 2024-09-21T01:29:12.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:12 smithi150 ceph-mon[34945]: pgmap v1198: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 10 MiB/s wr, 408 op/s 2024-09-21T01:29:12.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:12 smithi160 ceph-mon[34696]: pgmap v1198: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 10 MiB/s wr, 408 op/s 2024-09-21T01:29:12.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:12 smithi039 ceph-mon[30415]: pgmap v1198: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 10 MiB/s wr, 408 op/s 2024-09-21T01:29:13.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.02 MB/sec execute 579 sec latency 5498.397 ms 2024-09-21T01:29:14.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 14.00 MB/sec execute 580 sec latency 6498.519 ms 2024-09-21T01:29:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:14 smithi150 ceph-mon[34945]: pgmap v1199: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 10 MiB/s wr, 396 op/s 2024-09-21T01:29:14.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:14 smithi160 ceph-mon[34696]: pgmap v1199: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 10 MiB/s wr, 396 op/s 2024-09-21T01:29:14.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:14 smithi039 ceph-mon[30415]: pgmap v1199: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 10 MiB/s wr, 396 op/s 2024-09-21T01:29:15.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.97 MB/sec execute 581 sec latency 7498.579 ms 2024-09-21T01:29:16.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.95 MB/sec execute 582 sec latency 8498.743 ms 2024-09-21T01:29:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:15 smithi150 ceph-mon[34945]: pgmap v1200: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.5 MiB/s wr, 287 op/s 2024-09-21T01:29:16.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:15 smithi150 ceph-mon[34945]: osdmap e130: 12 total, 12 up, 12 in 2024-09-21T01:29:16.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:15 smithi160 ceph-mon[34696]: pgmap v1200: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.5 MiB/s wr, 287 op/s 2024-09-21T01:29:16.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:15 smithi160 ceph-mon[34696]: osdmap e130: 12 total, 12 up, 12 in 2024-09-21T01:29:16.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:15 smithi039 ceph-mon[30415]: pgmap v1200: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.5 MiB/s wr, 287 op/s 2024-09-21T01:29:16.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:15 smithi039 ceph-mon[30415]: osdmap e130: 12 total, 12 up, 12 in 2024-09-21T01:29:17.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.93 MB/sec execute 583 sec latency 9498.894 ms 2024-09-21T01:29:18.062 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.90 MB/sec execute 584 sec latency 10499.059 ms 2024-09-21T01:29:18.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:17 smithi150 ceph-mon[34945]: pgmap v1202: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.0 MiB/s wr, 241 op/s 2024-09-21T01:29:18.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:17 smithi160 ceph-mon[34696]: pgmap v1202: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.0 MiB/s wr, 241 op/s 2024-09-21T01:29:18.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:17 smithi039 ceph-mon[30415]: pgmap v1202: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.0 MiB/s wr, 241 op/s 2024-09-21T01:29:19.028 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:29:19.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.88 MB/sec execute 585 sec latency 11499.160 ms 2024-09-21T01:29:19.447 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:20.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:19 smithi039 ceph-mon[30415]: pgmap v1203: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:29:20.061 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.85 MB/sec execute 586 sec latency 12499.228 ms 2024-09-21T01:29:20.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:19 smithi150 ceph-mon[34945]: pgmap v1203: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:29:20.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:19 smithi160 ceph-mon[34696]: pgmap v1203: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:29:20.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.176+0000 7f34c4c00640 1 -- 172.21.15.39:0/4025402967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c010b6c0 msgr2=0x7f34c010bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:20.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.176+0000 7f34c4c00640 1 --2- 172.21.15.39:0/4025402967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c010b6c0 0x7f34c010bad0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009900 tx=0x7f34a802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 -- 172.21.15.39:0/4025402967 shutdown_connections 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 --2- 172.21.15.39:0/4025402967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34c010c010 0x7f34c0112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 --2- 172.21.15.39:0/4025402967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c010b6c0 0x7f34c010bad0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 --2- 172.21.15.39:0/4025402967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c0105d80 0x7f34c0106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 -- 172.21.15.39:0/4025402967 >> 172.21.15.39:0/4025402967 conn(0x7f34c01017d0 msgr2=0x7f34c0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 -- 172.21.15.39:0/4025402967 shutdown_connections 2024-09-21T01:29:20.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.177+0000 7f34c4c00640 1 -- 172.21.15.39:0/4025402967 wait complete. 2024-09-21T01:29:20.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 Processor -- start 2024-09-21T01:29:20.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 -- start start 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 0x7f34c019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c010b6c0 0x7f34c019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34c010c010 0x7f34c01a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0115d20 con 0x7f34c0105d80 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0115ba0 con 0x7f34c010c010 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.178+0000 7f34c4c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0115ea0 con 0x7f34c010b6c0 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 0x7f34c019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 0x7f34c019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53256/0 (socket says 172.21.15.39:53256) 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 -- 172.21.15.39:0/1241378133 learned_addr learned my addr 172.21.15.39:0/1241378133 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34b7e00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c010b6c0 0x7f34c019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34be800640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34c010c010 0x7f34c01a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 -- 172.21.15.39:0/1241378133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c010b6c0 msgr2=0x7f34c019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:20.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c010b6c0 0x7f34c019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 -- 172.21.15.39:0/1241378133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34c010c010 msgr2=0x7f34c01a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:20.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34c010c010 0x7f34c01a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.179+0000 7f34bde00640 1 -- 172.21.15.39:0/1241378133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c01a2640 con 0x7f34c0105d80 2024-09-21T01:29:20.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.180+0000 7f34bde00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 0x7f34c019c6d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f34b000e9f0 tx=0x7f34b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.180+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b000cd80 con 0x7f34c0105d80 2024-09-21T01:29:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.180+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00054d0 con 0x7f34c0105d80 2024-09-21T01:29:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.180+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c01a2930 con 0x7f34c0105d80 2024-09-21T01:29:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.180+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0010690 con 0x7f34c0105d80 2024-09-21T01:29:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.180+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c01a9090 con 0x7f34c0105d80 2024-09-21T01:29:20.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.181+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c0106190 con 0x7f34c0105d80 2024-09-21T01:29:20.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.186+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f34b0004e30 con 0x7f34c0105d80 2024-09-21T01:29:20.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.187+0000 7f34b6a00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f34a40b6680 0x7f34a40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:20.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.187+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f34b00e8110 con 0x7f34c0105d80 2024-09-21T01:29:20.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.187+0000 7f34b7e00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f34a40b6680 0x7f34a40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:20.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.188+0000 7f34b7e00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f34a40b6680 0x7f34a40b8b90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f34a802a9e0 tx=0x7f34a8005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:20.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.189+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f34b00ae220 con 0x7f34c0105d80 2024-09-21T01:29:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.427+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34c019d800 con 0x7f34c0105d80 2024-09-21T01:29:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.428+0000 7f34b6a00640 1 -- 172.21.15.39:0/1241378133 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+11912 (secure 0 0 0) 0x7f34c019d800 con 0x7f34c0105d80 2024-09-21T01:29:20.429 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:29:20.429 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":66,"btime":"2024-09-21T01:29:02:973821+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":66,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:29:02.196351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:29:20.429 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 66 2024-09-21T01:29:20.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f34a40b6680 msgr2=0x7f34a40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:20.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f34a40b6680 0x7f34a40b8b90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f34a802a9e0 tx=0x7f34a8005a20 comp rx=0 tx=0).stop 2024-09-21T01:29:20.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 msgr2=0x7f34c019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:20.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 0x7f34c019c6d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f34b000e9f0 tx=0x7f34b000edf0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 shutdown_connections 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f34a40b6680 0x7f34a40b8b90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34c010c010 0x7f34c01a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f34c010b6c0 0x7f34c019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 --2- 172.21.15.39:0/1241378133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34c0105d80 0x7f34c019c6d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 >> 172.21.15.39:0/1241378133 conn(0x7f34c01017d0 msgr2=0x7f34c0103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 shutdown_connections 2024-09-21T01:29:20.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:20.431+0000 7f34c4c00640 1 -- 172.21.15.39:0/1241378133 wait complete. 2024-09-21T01:29:21.062 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.83 MB/sec execute 587 sec latency 13499.360 ms 2024-09-21T01:29:21.063 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:20 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1241378133' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:21.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:20 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1241378133' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:21.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:20 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1241378133' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:21.146 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --set-id ff810f0f --release --await' 2024-09-21T01:29:21.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --set-id ff810f0f --release --await 2024-09-21T01:29:21.568 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:22.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:21 smithi039 ceph-mon[30415]: pgmap v1204: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-09-21T01:29:22.062 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186500 13.81 MB/sec execute 588 sec latency 14499.481 ms 2024-09-21T01:29:22.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:21 smithi150 ceph-mon[34945]: pgmap v1204: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-09-21T01:29:22.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:21 smithi160 ceph-mon[34696]: pgmap v1204: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 -- 172.21.15.39:0/3665444948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0077380 msgr2=0x7ff3a00757b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 --2- 172.21.15.39:0/3665444948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0077380 0x7ff3a00757b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff390009900 tx=0x7ff39002a400 comp rx=0 tx=0).stop 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 -- 172.21.15.39:0/3665444948 shutdown_connections 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 --2- 172.21.15.39:0/3665444948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0114430 0x7ff3a0116900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 --2- 172.21.15.39:0/3665444948 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 0x7ff3a00761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 --2- 172.21.15.39:0/3665444948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0077380 0x7ff3a00757b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 -- 172.21.15.39:0/3665444948 >> 172.21.15.39:0/3665444948 conn(0x7ff3a007b850 msgr2=0x7ff3a007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:22.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 -- 172.21.15.39:0/3665444948 shutdown_connections 2024-09-21T01:29:22.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.209+0000 7ff3a5400640 1 -- 172.21.15.39:0/3665444948 wait complete. 2024-09-21T01:29:22.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.211+0000 7ff3a5400640 1 Processor -- start 2024-09-21T01:29:22.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 -- start start 2024-09-21T01:29:22.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 0x7ff3a01a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 0x7ff3a01a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0114430 0x7ff3a01a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a011a0b0 con 0x7ff3a0114430 2024-09-21T01:29:22.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a0119f30 con 0x7ff3a0077380 2024-09-21T01:29:22.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff3a5400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a011a230 con 0x7ff3a0075dd0 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff397e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 0x7ff3a01a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff39e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 0x7ff3a01a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff397e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 0x7ff3a01a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38924/0 (socket says 172.21.15.39:38924) 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff397e00640 1 -- 172.21.15.39:0/4030513787 learned_addr learned my addr 172.21.15.39:0/4030513787 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.212+0000 7ff39e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 0x7ff3a01a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:60264/0 (socket says 172.21.15.39:60264) 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff39f200640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0114430 0x7ff3a01a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff397e00640 1 -- 172.21.15.39:0/4030513787 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 msgr2=0x7ff3a01a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff397e00640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 0x7ff3a01a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff397e00640 1 -- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0114430 msgr2=0x7ff3a01a62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff397e00640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0114430 0x7ff3a01a62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff397e00640 1 -- 172.21.15.39:0/4030513787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a01a69f0 con 0x7ff3a0077380 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.213+0000 7ff397e00640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 0x7ff3a01a1020 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff38800e9a0 tx=0x7ff38800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.214+0000 7ff397400640 1 -- 172.21.15.39:0/4030513787 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38800cd80 con 0x7ff3a0077380 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.214+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a01a6ce0 con 0x7ff3a0077380 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.214+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a01ad440 con 0x7ff3a0077380 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.214+0000 7ff397400640 1 -- 172.21.15.39:0/4030513787 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3880035f0 con 0x7ff3a0077380 2024-09-21T01:29:22.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.214+0000 7ff397400640 1 -- 172.21.15.39:0/4030513787 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388010690 con 0x7ff3a0077380 2024-09-21T01:29:22.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.218+0000 7ff397400640 1 -- 172.21.15.39:0/4030513787 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff388003aa0 con 0x7ff3a0077380 2024-09-21T01:29:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 Processor -- start 2024-09-21T01:29:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 -- start start 2024-09-21T01:29:22.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58232b520 0x55e5826d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58230a840 0x55e5826d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e582649290 con 0x55e58232b520 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e5826386c0 con 0x55e582302660 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.219+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e58263c4d0 con 0x55e58230a840 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58232b520 0x55e5826d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826d2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38934/0 (socket says 172.21.15.39:38934) 2024-09-21T01:29:22.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58232b520 0x55e5826d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53278/0 (socket says 172.21.15.39:53278) 2024-09-21T01:29:22.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 -- 172.21.15.39:0/3580299484 learned_addr learned my addr 172.21.15.39:0/3580299484 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:22.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff397400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3840b6700 0x7ff3840b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff397e00640 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58230a840 0x55e5826d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 -- 172.21.15.39:0/3580299484 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58230a840 msgr2=0x55e5826d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58230a840 0x55e5826d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 -- 172.21.15.39:0/3580299484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58232b520 msgr2=0x55e5826d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58232b520 0x55e5826d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39e800640 1 -- 172.21.15.39:0/3580299484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5825c5460 con 0x55e582302660 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff39f200640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3840b6700 0x7ff3840b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.220+0000 7ff397400640 1 -- 172.21.15.39:0/4030513787 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7ff3880e7da0 con 0x7ff3a0077380 2024-09-21T01:29:22.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.221+0000 7ff39e800640 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826d2700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff390009230 tx=0x7ff390004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81510d095b1bafca server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.221+0000 7ff39f200640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3840b6700 0x7ff3840b8c10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff398005fd0 tx=0x7ff398004ab0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.222+0000 7ff38f400640 1 -- 172.21.15.39:0/3580299484 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390035040 con 0x55e582302660 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.222+0000 7ff38f400640 1 -- 172.21.15.39:0/3580299484 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3900054f0 con 0x55e582302660 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.222+0000 7ff38f400640 1 -- 172.21.15.39:0/3580299484 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3900035f0 con 0x55e582302660 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/3580299484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 msgr2=0x55e5826d2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826d2700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff390009230 tx=0x7ff390004040 comp rx=0 tx=0).stop 2024-09-21T01:29:22.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/3580299484 shutdown_connections 2024-09-21T01:29:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58230a840 0x55e5826d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826d2700 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/3580299484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58232b520 0x55e5826d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/3580299484 >> 172.21.15.39:0/3580299484 conn(0x55e58231f910 msgr2=0x55e5826d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/3580299484 shutdown_connections 2024-09-21T01:29:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.223+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/3580299484 wait complete. 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 -- start start 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 Processor -- start 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826dddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826de2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58232b520 0x55e5826e9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e582499bb0 con 0x55e582302660 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e5825d36c0 con 0x55e58230a840 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x55e5826e0940 con 0x55e58232b520 2024-09-21T01:29:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.224+0000 7ff3ac1d3740 10 client.0 _ll_register_callbacks cb 0x55e5825ee490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:29:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826de2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff39f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58232b520 0x55e5826e9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826de2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38936/0 (socket says 172.21.15.39:38936) 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff39e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826dddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff39f200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58232b520 0x55e5826e9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:60286/0 (socket says 172.21.15.39:60286) 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 -- 172.21.15.39:0/2352068714 learned_addr learned my addr 172.21.15.39:0/2352068714 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 -- 172.21.15.39:0/2352068714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58232b520 msgr2=0x55e5826e9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58232b520 0x55e5826e9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 -- 172.21.15.39:0/2352068714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 msgr2=0x55e5826dddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826dddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff397e00640 1 -- 172.21.15.39:0/2352068714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e582396b00 con 0x55e58230a840 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.225+0000 7ff39e800640 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826dddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff397e00640 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826de2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff388000c00 tx=0x7ff3880095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed9be2ff7443d3b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff383e00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff383e00640 1 -- 172.21.15.39:0/2352068714 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3880ad2c0 con 0x55e58230a840 2024-09-21T01:29:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3900054f0 con 0x55e58230a840 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff383e00640 1 -- 172.21.15.39:0/2352068714 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7ff3880aeae0 con 0x55e58230a840 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff383e00640 1 -- 172.21.15.39:0/2352068714 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7ff3880adeb0 con 0x55e58230a840 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff3ac1d3740 10 client.26609 fetch_fsmap learned FSMap version 66 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.226+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55e58240d960 con 0x55e58230a840 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff383e00640 1 -- 172.21.15.39:0/2352068714 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3900054f0 con 0x55e58230a840 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff383e00640 1 -- 172.21.15.39:0/2352068714 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 66) ==== 8358+0+0 (secure 0 0 0) 0x7ff3880ac480 con 0x55e58230a840 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff3ac1d3740 10 client.26609 fetch_fsmap finished waiting for FSMap version 66 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff3ac1d3740 10 client.26609 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff3ac1d3740 20 client.26609 populate_metadata read hostname 'smithi039' 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e5822e9ea0 0x55e5826efd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff3ac1d3740 4 client.26609 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.227+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e58264b1e0 con 0x55e5822e9ea0 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.228+0000 7ff39e800640 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e5822e9ea0 0x55e5826efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.228+0000 7ff39e800640 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e5822e9ea0 0x55e5826efd60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50a7e6cf3bdbd86 server_cookie=cee1a31043adb752 in_seq=0 out_seq=0 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.228+0000 7ff383e00640 10 client.26609 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff383e00640 1 -- 172.21.15.39:0/2352068714 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e58264b1e0 con 0x55e5822e9ea0 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff383e00640 10 client.26609 handle_command_reply: tid=0 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff3ac1d3740 1 client.26609 shutdown 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 msgr2=0x55e5826de2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826de2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff388000c00 tx=0x7ff3880095b0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 shutdown_connections 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.229+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e5822e9ea0 0x55e5826efd60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e58232b520 0x55e5826e9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826de2f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff383e00640 0 client.26609 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2352068714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826dddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 >> 172.21.15.39:0/2352068714 conn(0x55e58231f910 msgr2=0x55e5826d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 shutdown_connections 2024-09-21T01:29:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2352068714 wait complete. 2024-09-21T01:29:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.230+0000 7ff3ac1d3740 20 client.26609 trim_cache size 0 max 16384 2024-09-21T01:29:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 Processor -- start 2024-09-21T01:29:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 -- start start 2024-09-21T01:29:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826dada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01b9ee0 con 0x55e582302660 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01b7780 con 0x55e58230a840 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.248+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01b55f0 con 0x55e5822e9ea0 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff397e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:60298/0 (socket says 172.21.15.39:60298) 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 -- 172.21.15.39:0/2577005072 learned_addr learned my addr 172.21.15.39:0/2577005072 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39f200640 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826dada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 -- 172.21.15.39:0/2577005072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 msgr2=0x55e5826dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826dada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 -- 172.21.15.39:0/2577005072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 msgr2=0x55e5826cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 -- 172.21.15.39:0/2577005072 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5826dc0b0 con 0x55e5822e9ea0 2024-09-21T01:29:22.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.249+0000 7ff39e800640 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826cf660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff390004b90 tx=0x7ff390004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd3191540db50f2f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:22.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.250+0000 7ff38ea00640 1 -- 172.21.15.39:0/2577005072 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390035040 con 0x55e5822e9ea0 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.250+0000 7ff38ea00640 1 -- 172.21.15.39:0/2577005072 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff39005c070 con 0x55e5822e9ea0 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.250+0000 7ff38ea00640 1 -- 172.21.15.39:0/2577005072 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390003850 con 0x55e5822e9ea0 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.250+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2577005072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 msgr2=0x55e5826cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.250+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826cf660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff390004b90 tx=0x7ff390004b20 comp rx=0 tx=0).stop 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2577005072 shutdown_connections 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e58230a840 0x55e5826dada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e582302660 0x55e5826cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2577005072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826cf660 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2577005072 >> 172.21.15.39:0/2577005072 conn(0x55e58231f910 msgr2=0x55e5826ea220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2577005072 shutdown_connections 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2577005072 wait complete. 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 -- start start 2024-09-21T01:29:22.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.251+0000 7ff3ac1d3740 1 Processor -- start 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826e6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826e2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58230a840 0x55e5826e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01b5c40 con 0x55e58230a840 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01b70b0 con 0x55e582302660 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01b6310 con 0x55e5822e9ea0 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff3ac1d3740 10 client.0 _ll_register_callbacks cb 0x7ff38800ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff397e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff39e800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826e6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff39f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58230a840 0x55e5826e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff397e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826e2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:38954/0 (socket says 172.21.15.39:38954) 2024-09-21T01:29:22.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.252+0000 7ff397e00640 1 -- 172.21.15.39:0/2465999045 learned_addr learned my addr 172.21.15.39:0/2465999045 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:22.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff39e800640 1 -- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 msgr2=0x55e5826e2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff39e800640 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826e2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff39e800640 1 -- 172.21.15.39:0/2465999045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58230a840 msgr2=0x55e5826e2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff39e800640 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58230a840 0x55e5826e2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff39e800640 1 -- 172.21.15.39:0/2465999045 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5826e3290 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff39e800640 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826e6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff3900057c0 tx=0x7ff390031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e4a966ccca6cec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff383e00640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff383e00640 1 -- 172.21.15.39:0/2465999045 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390066070 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff39005c070 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.253+0000 7ff383e00640 1 -- 172.21.15.39:0/2465999045 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7ff390004c30 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff383e00640 1 -- 172.21.15.39:0/2465999045 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7ff390072050 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 10 client.26587 fetch_fsmap learned FSMap version 66 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55e58275b560 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff383e00640 1 -- 172.21.15.39:0/2465999045 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff39005c070 con 0x55e5822e9ea0 2024-09-21T01:29:22.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff383e00640 1 -- 172.21.15.39:0/2465999045 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 66) ==== 8358+0+0 (secure 0 0 0) 0x7ff390003aa0 con 0x55e5822e9ea0 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 10 client.26587 fetch_fsmap finished waiting for FSMap version 66 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 10 client.26587 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 20 client.26587 populate_metadata read hostname 'smithi039' 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e58232b520 0x55e58275d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 4 client.26587 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ff810f0f", "release": true, "await": true}] 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.254+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ff810f0f", "release": true, "await": true}) -- 0x55e582646c90 con 0x55e58232b520 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.255+0000 7ff397e00640 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e58232b520 0x55e58275d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.255+0000 7ff397e00640 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e58232b520 0x55e58275d8a0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2563a52ccc542e03 server_cookie=872a710eca216fe1 in_seq=0 out_seq=0 2024-09-21T01:29:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.255+0000 7ff383e00640 10 client.26587 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:22.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.295+0000 7ff383e00640 1 -- 172.21.15.39:0/2465999045 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+593 (crc 0 0 0) 0x55e582646c90 con 0x55e58232b520 2024-09-21T01:29:22.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.295+0000 7ff383e00640 10 client.26587 handle_command_reply: tid=0 2024-09-21T01:29:22.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.295+0000 7ff3ac1d3740 1 client.26587 shutdown 2024-09-21T01:29:22.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.295+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 msgr2=0x55e5826e6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.295+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826e6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff3900057c0 tx=0x7ff390031400 comp rx=0 tx=0).stop 2024-09-21T01:29:22.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 shutdown_connections 2024-09-21T01:29:22.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x55e58232b520 0x55e58275d8a0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e58230a840 0x55e5826e2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e582302660 0x55e5826e2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff3ac1d3740 1 --2- 172.21.15.39:0/2465999045 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x55e5822e9ea0 0x55e5826e6250 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff383e00640 0 client.26587 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.296+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 >> 172.21.15.39:0/2465999045 conn(0x55e58231f910 msgr2=0x55e5826df010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.297+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 shutdown_connections 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.297+0000 7ff3ac1d3740 1 -- 172.21.15.39:0/2465999045 wait complete. 2024-09-21T01:29:22.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.297+0000 7ff3ac1d3740 20 client.26587 trim_cache size 0 max 16384 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 66, 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 100, 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": { 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "ff810f0f": { 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "version": 100, 2024-09-21T01:29:22.298 INFO:teuthology.orchestra.run.smithi039.stdout: "age_ref": 0.0, 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "timeout": 90.0, 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "expiration": 139.8, 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "members": { 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "file:/": { 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "excluded": false, 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "state": { 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RELEASED", 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.0 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:22.299 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:22.300 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:22.300 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:22.300 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:22.300 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:29:22.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3840b6700 msgr2=0x7ff3840b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3840b6700 0x7ff3840b8c10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff398005fd0 tx=0x7ff398004ab0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 msgr2=0x7ff3a01a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 0x7ff3a01a1020 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff38800e9a0 tx=0x7ff38800ee60 comp rx=0 tx=0).stop 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 shutdown_connections 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3a0114430 0x7ff3a01a62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3840b6700 0x7ff3840b8c10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a0077380 0x7ff3a01a1020 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 --2- 172.21.15.39:0/4030513787 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3a0075dd0 0x7ff3a01a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 >> 172.21.15.39:0/4030513787 conn(0x7ff3a007b850 msgr2=0x7ff3a0113030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.298+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 shutdown_connections 2024-09-21T01:29:22.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:22.299+0000 7ff3a5400640 1 -- 172.21.15.39:0/4030513787 wait complete. 2024-09-21T01:29:23.064 INFO:tasks.workunit.client.0.smithi039.stdout: 10 186821 13.81 MB/sec execute 589 sec latency 14698.865 ms 2024-09-21T01:29:23.139 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ff810f0f', total seconds elapsed: 23.9 2024-09-21T01:29:24.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:23 smithi150 ceph-mon[34945]: pgmap v1205: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 467 KiB/s wr, 25 op/s 2024-09-21T01:29:24.064 INFO:tasks.workunit.client.0.smithi039.stdout: 10 187275 13.83 MB/sec execute 590 sec latency 154.687 ms 2024-09-21T01:29:24.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:23 smithi160 ceph-mon[34696]: pgmap v1205: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 467 KiB/s wr, 25 op/s 2024-09-21T01:29:24.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:23 smithi039 ceph-mon[30415]: pgmap v1205: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 467 KiB/s wr, 25 op/s 2024-09-21T01:29:25.064 INFO:tasks.workunit.client.0.smithi039.stdout: 10 187576 13.82 MB/sec execute 591 sec latency 162.225 ms 2024-09-21T01:29:26.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:25 smithi150 ceph-mon[34945]: pgmap v1206: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 568 KiB/s wr, 35 op/s 2024-09-21T01:29:26.064 INFO:tasks.workunit.client.0.smithi039.stdout: 10 187888 13.82 MB/sec execute 592 sec latency 464.787 ms 2024-09-21T01:29:26.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:25 smithi160 ceph-mon[34696]: pgmap v1206: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 568 KiB/s wr, 35 op/s 2024-09-21T01:29:26.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:25 smithi039 ceph-mon[30415]: pgmap v1206: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 568 KiB/s wr, 35 op/s 2024-09-21T01:29:27.066 INFO:tasks.workunit.client.0.smithi039.stdout: 10 188470 13.86 MB/sec execute 593 sec latency 159.429 ms 2024-09-21T01:29:28.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:27 smithi150 ceph-mon[34945]: pgmap v1207: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 514 KiB/s wr, 34 op/s 2024-09-21T01:29:28.065 INFO:tasks.workunit.client.0.smithi039.stdout: 10 189036 13.89 MB/sec execute 594 sec latency 121.015 ms 2024-09-21T01:29:28.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:27 smithi160 ceph-mon[34696]: pgmap v1207: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 514 KiB/s wr, 34 op/s 2024-09-21T01:29:28.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:27 smithi039 ceph-mon[30415]: pgmap v1207: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 514 KiB/s wr, 34 op/s 2024-09-21T01:29:29.066 INFO:tasks.workunit.client.0.smithi039.stdout: 10 189597 13.93 MB/sec execute 595 sec latency 136.131 ms 2024-09-21T01:29:29.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:29 smithi039 ceph-mon[30415]: pgmap v1208: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.1 MiB/s wr, 187 op/s 2024-09-21T01:29:30.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:29 smithi150 ceph-mon[34945]: pgmap v1208: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.1 MiB/s wr, 187 op/s 2024-09-21T01:29:30.066 INFO:tasks.workunit.client.0.smithi039.stdout: 10 190037 13.94 MB/sec execute 596 sec latency 103.290 ms 2024-09-21T01:29:30.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:29 smithi160 ceph-mon[34696]: pgmap v1208: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.1 MiB/s wr, 187 op/s 2024-09-21T01:29:31.067 INFO:tasks.workunit.client.0.smithi039.stdout: 10 190458 13.95 MB/sec execute 597 sec latency 107.299 ms 2024-09-21T01:29:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:31 smithi039 ceph-mon[30415]: pgmap v1209: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.9 MiB/s wr, 228 op/s 2024-09-21T01:29:32.067 INFO:tasks.workunit.client.0.smithi039.stdout: 10 190861 13.96 MB/sec execute 598 sec latency 307.347 ms 2024-09-21T01:29:32.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:31 smithi160 ceph-mon[34696]: pgmap v1209: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.9 MiB/s wr, 228 op/s 2024-09-21T01:29:32.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:31 smithi150 ceph-mon[34945]: pgmap v1209: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.9 MiB/s wr, 228 op/s 2024-09-21T01:29:33.067 INFO:tasks.workunit.client.0.smithi039.stdout: 10 191411 13.99 MB/sec execute 599 sec latency 107.844 ms 2024-09-21T01:29:33.819 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:33 smithi039 ceph-mon[30415]: pgmap v1210: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.3 MiB/s wr, 326 op/s 2024-09-21T01:29:34.067 INFO:tasks.workunit.client.0.smithi039.stdout: 10 cleanup 600 sec 2024-09-21T01:29:34.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:33 smithi160 ceph-mon[34696]: pgmap v1210: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.3 MiB/s wr, 326 op/s 2024-09-21T01:29:34.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:33 smithi150 ceph-mon[34945]: pgmap v1210: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.3 MiB/s wr, 326 op/s 2024-09-21T01:29:35.067 INFO:tasks.workunit.client.0.smithi039.stdout: 1 cleanup 601 sec 2024-09-21T01:29:35.233 INFO:tasks.workunit.client.0.smithi039.stdout: 0 cleanup 601 sec 2024-09-21T01:29:35.233 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-21T01:29:35.233 INFO:tasks.workunit.client.0.smithi039.stdout: Operation Count AvgLat MaxLat 2024-09-21T01:29:35.233 INFO:tasks.workunit.client.0.smithi039.stdout: ---------------------------------------- 2024-09-21T01:29:35.233 INFO:tasks.workunit.client.0.smithi039.stdout: NTCreateX 269239 8.734 33451.244 2024-09-21T01:29:35.233 INFO:tasks.workunit.client.0.smithi039.stdout: Close 197805 0.007 2.293 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Rename 11438 32.796 33318.485 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Unlink 54370 14.725 29676.564 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Qpathinfo 244332 6.441 33786.771 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Qfileinfo 42654 0.004 28.255 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Qfsinfo 44843 0.919 3032.562 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Sfileinfo 21984 2.863 14622.821 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Find 94521 1.723 18558.083 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: WriteX 133293 1.673 18570.365 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: ReadX 423427 0.108 3373.161 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: LockX 880 3.276 306.778 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: UnlockX 880 4.647 2007.969 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: Flush 18890 18.551 16161.136 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-21T01:29:35.234 INFO:tasks.workunit.client.0.smithi039.stdout:Throughput 13.9857 MB/sec 10 clients 10 procs max_latency=33786.782 ms 2024-09-21T01:29:35.253 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T01:29:35.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T01:29:36.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:35 smithi160 ceph-mon[34696]: pgmap v1211: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 379 op/s 2024-09-21T01:29:36.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:35 smithi039 ceph-mon[30415]: pgmap v1211: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 379 op/s 2024-09-21T01:29:36.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:35 smithi150 ceph-mon[34945]: pgmap v1211: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 379 op/s 2024-09-21T01:29:37.247 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-09-21T01:29:37.247 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T01:29:38.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:37 smithi160 ceph-mon[34696]: pgmap v1212: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 372 op/s 2024-09-21T01:29:38.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:37 smithi039 ceph-mon[30415]: pgmap v1212: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 372 op/s 2024-09-21T01:29:38.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:37 smithi150 ceph-mon[34945]: pgmap v1212: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.5 MiB/s wr, 372 op/s 2024-09-21T01:29:38.410 DEBUG:teuthology.parallel:result is None 2024-09-21T01:29:38.410 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T01:29:38.482 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T01:29:38.482 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-21T01:29:38.498 INFO:tasks.quiescer:joining Quiescers 2024-09-21T01:29:38.498 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-21T01:29:38.498 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:29:38.916 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:39.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:38 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:39.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:38 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:39.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:38 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:39.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 -- 172.21.15.39:0/3442012065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c109df0 msgr2=0x7f411c110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:39.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 --2- 172.21.15.39:0/3442012065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c109df0 0x7f411c110620 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4114009ca0 tx=0x7f411402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 -- 172.21.15.39:0/3442012065 shutdown_connections 2024-09-21T01:29:39.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 --2- 172.21.15.39:0/3442012065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c109df0 0x7f411c110620 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 --2- 172.21.15.39:0/3442012065 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f411c1094a0 0x7f411c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 --2- 172.21.15.39:0/3442012065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f411c103b60 0x7f411c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 -- 172.21.15.39:0/3442012065 >> 172.21.15.39:0/3442012065 conn(0x7f411c07b850 msgr2=0x7f411c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:39.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 -- 172.21.15.39:0/3442012065 shutdown_connections 2024-09-21T01:29:39.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.639+0000 7f4122a00640 1 -- 172.21.15.39:0/3442012065 wait complete. 2024-09-21T01:29:39.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.640+0000 7f4122a00640 1 Processor -- start 2024-09-21T01:29:39.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.640+0000 7f4122a00640 1 -- start start 2024-09-21T01:29:39.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.640+0000 7f4122a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 0x7f411c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:39.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.640+0000 7f4122a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f411c1094a0 0x7f411c19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.640+0000 7f4122a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f411c109df0 0x7f411c1a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.640+0000 7f4122a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f411c113ab0 con 0x7f411c103b60 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f4122a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f411c113930 con 0x7f411c109df0 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f4122a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f411c113c30 con 0x7f411c1094a0 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 0x7f411c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 0x7f411c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38272/0 (socket says 172.21.15.39:38272) 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 -- 172.21.15.39:0/1182543047 learned_addr learned my addr 172.21.15.39:0/1182543047 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411b400640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f411c1094a0 0x7f411c19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f4120a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f411c109df0 0x7f411c1a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 -- 172.21.15.39:0/1182543047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f411c1094a0 msgr2=0x7f411c19cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f411c1094a0 0x7f411c19cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 -- 172.21.15.39:0/1182543047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f411c109df0 msgr2=0x7f411c1a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:39.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f411c109df0 0x7f411c1a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.641+0000 7f411be00640 1 -- 172.21.15.39:0/1182543047 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c1a25f0 con 0x7f411c103b60 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.642+0000 7f411be00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 0x7f411c19c680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f410c00e9f0 tx=0x7f410c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.642+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c00cd80 con 0x7f411c103b60 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.642+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f410c0108e0 con 0x7f411c103b60 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.642+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c018980 con 0x7f411c103b60 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.642+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411c1a28e0 con 0x7f411c103b60 2024-09-21T01:29:39.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.642+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411c1a9040 con 0x7f411c103b60 2024-09-21T01:29:39.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.643+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f411c103f70 con 0x7f411c103b60 2024-09-21T01:29:39.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.646+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f410c010430 con 0x7f411c103b60 2024-09-21T01:29:39.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.647+0000 7f4118c00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41000b6700 0x7f41000b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:39.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.647+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f410c0e7ec0 con 0x7f411c103b60 2024-09-21T01:29:39.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.647+0000 7f411b400640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41000b6700 0x7f41000b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:39.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.648+0000 7f411b400640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41000b6700 0x7f41000b8c10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f411c106450 tx=0x7f4108005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:39.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.651+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f410c0adfd0 con 0x7f411c103b60 2024-09-21T01:29:39.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.899+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f411c19d7b0 con 0x7f411c103b60 2024-09-21T01:29:39.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.900+0000 7f4118c00640 1 -- 172.21.15.39:0/1182543047 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+11915 (secure 0 0 0) 0x7f410c0b23b0 con 0x7f411c103b60 2024-09-21T01:29:39.901 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:29:39.902 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":68,"btime":"2024-09-21T01:29:39:840404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":68,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:29:39.419575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:29:39.902 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 68 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.902+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41000b6700 msgr2=0x7f41000b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.902+0000 7f4122a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41000b6700 0x7f41000b8c10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f411c106450 tx=0x7f4108005f90 comp rx=0 tx=0).stop 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.902+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 msgr2=0x7f411c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.902+0000 7f4122a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 0x7f411c19c680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f410c00e9f0 tx=0x7f410c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 shutdown_connections 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f41000b6700 0x7f41000b8c10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f411c109df0 0x7f411c1a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f411c1094a0 0x7f411c19cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 --2- 172.21.15.39:0/1182543047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f411c103b60 0x7f411c19c680 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 >> 172.21.15.39:0/1182543047 conn(0x7f411c07b850 msgr2=0x7f411c1072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 shutdown_connections 2024-09-21T01:29:39.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:39.903+0000 7f4122a00640 1 -- 172.21.15.39:0/1182543047 wait complete. 2024-09-21T01:29:40.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:39 smithi160 ceph-mon[34696]: pgmap v1213: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 464 op/s 2024-09-21T01:29:40.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:39 smithi039 ceph-mon[30415]: pgmap v1213: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 464 op/s 2024-09-21T01:29:40.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:39 smithi150 ceph-mon[34945]: pgmap v1213: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 464 op/s 2024-09-21T01:29:40.666 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24764 quiesce db --cancel --all' 2024-09-21T01:29:40.666 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24764 quiesce db --cancel --all 2024-09-21T01:29:40.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:40 smithi039 ceph-mon[30415]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:40.903 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:40 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1182543047' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:41.051 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:41.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:40 smithi160 ceph-mon[34696]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:41.107 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:40 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1182543047' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:40 smithi150 ceph-mon[34945]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-21T01:29:41.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:40 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1182543047' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:29:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 -- 172.21.15.39:0/3526433496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c105d80 msgr2=0x7f352c106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 --2- 172.21.15.39:0/3526433496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c105d80 0x7f352c106190 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f351c009900 tx=0x7f351c02a410 comp rx=0 tx=0).stop 2024-09-21T01:29:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 -- 172.21.15.39:0/3526433496 shutdown_connections 2024-09-21T01:29:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 --2- 172.21.15.39:0/3526433496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c10c010 0x7f352c112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 --2- 172.21.15.39:0/3526433496 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f352c10b6c0 0x7f352c10bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 --2- 172.21.15.39:0/3526433496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c105d80 0x7f352c106190 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 -- 172.21.15.39:0/3526433496 >> 172.21.15.39:0/3526433496 conn(0x7f352c101810 msgr2=0x7f352c103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:41.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 -- 172.21.15.39:0/3526433496 shutdown_connections 2024-09-21T01:29:41.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.738+0000 7f3532a00640 1 -- 172.21.15.39:0/3526433496 wait complete. 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.739+0000 7f3532a00640 1 Processor -- start 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 -- start start 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 0x7f352c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f352c10b6c0 0x7f352c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c10c010 0x7f352c1a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f352c115d20 con 0x7f352c10c010 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f352c115ba0 con 0x7f352c105d80 2024-09-21T01:29:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3532a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f352c115ea0 con 0x7f352c10b6c0 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f352be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 0x7f352c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f352be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 0x7f352c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:46746/0 (socket says 172.21.15.39:46746) 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f352be00640 1 -- 172.21.15.39:0/826898191 learned_addr learned my addr 172.21.15.39:0/826898191 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3530a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c10c010 0x7f352c1a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.740+0000 7f3523400640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f352c10b6c0 0x7f352c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.741+0000 7f352be00640 1 -- 172.21.15.39:0/826898191 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f352c10b6c0 msgr2=0x7f352c19cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.741+0000 7f352be00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f352c10b6c0 0x7f352c19cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.741+0000 7f352be00640 1 -- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c10c010 msgr2=0x7f352c1a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.741+0000 7f352be00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c10c010 0x7f352c1a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.741+0000 7f352be00640 1 -- 172.21.15.39:0/826898191 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352c1a2660 con 0x7f352c105d80 2024-09-21T01:29:41.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.741+0000 7f352be00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 0x7f352c19c6f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f351c005ab0 tx=0x7f351c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:41.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.742+0000 7f3529600640 1 -- 172.21.15.39:0/826898191 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c03f070 con 0x7f352c105d80 2024-09-21T01:29:41.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.742+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352c1a28f0 con 0x7f352c105d80 2024-09-21T01:29:41.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.742+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352c1a90b0 con 0x7f352c105d80 2024-09-21T01:29:41.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.742+0000 7f3529600640 1 -- 172.21.15.39:0/826898191 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f351c031b60 con 0x7f352c105d80 2024-09-21T01:29:41.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.742+0000 7f3529600640 1 -- 172.21.15.39:0/826898191 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c049060 con 0x7f352c105d80 2024-09-21T01:29:41.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 Processor -- start 2024-09-21T01:29:41.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 -- start start 2024-09-21T01:29:41.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60eb6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60eb67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 0x558e60eb6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558e60e2dff0 con 0x558e60aef5f0 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558e60e1d420 con 0x558e60ae7410 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f353981a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558e60e21230 con 0x558e60b102d0 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.745+0000 7f3523400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 0x558e60eb6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3530a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60eb6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 0x558e60eb6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38312/0 (socket says 172.21.15.39:38312) 2024-09-21T01:29:41.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 -- 172.21.15.39:0/845956017 learned_addr learned my addr 172.21.15.39:0/845956017 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 -- 172.21.15.39:0/845956017 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 msgr2=0x558e60eb6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60eb6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 -- 172.21.15.39:0/845956017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 msgr2=0x558e60eb67d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60eb67d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 -- 172.21.15.39:0/845956017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e60b7b8b0 con 0x558e60aef5f0 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3523400640 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 0x558e60eb6d10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3514009c20 tx=0x7f3514011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa51a22901c95d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.746+0000 7f3529600640 1 -- 172.21.15.39:0/826898191 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f351c042070 con 0x7f352c105d80 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.747+0000 7f3521200640 1 -- 172.21.15.39:0/845956017 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514004230 con 0x558e60aef5f0 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.747+0000 7f3521200640 1 -- 172.21.15.39:0/845956017 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35140043d0 con 0x558e60aef5f0 2024-09-21T01:29:41.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.747+0000 7f3521200640 1 -- 172.21.15.39:0/845956017 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35140196d0 con 0x558e60aef5f0 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 -- 172.21.15.39:0/845956017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 msgr2=0x558e60eb6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 0x558e60eb6d10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3514009c20 tx=0x7f3514011810 comp rx=0 tx=0).stop 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 -- 172.21.15.39:0/845956017 shutdown_connections 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60aef5f0 0x558e60eb6d10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60eb67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 --2- 172.21.15.39:0/845956017 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60eb6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 -- 172.21.15.39:0/845956017 >> 172.21.15.39:0/845956017 conn(0x558e60b046c0 msgr2=0x558e60eb4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:41.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 -- 172.21.15.39:0/845956017 shutdown_connections 2024-09-21T01:29:41.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f353981a740 1 -- 172.21.15.39:0/845956017 wait complete. 2024-09-21T01:29:41.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f3529600640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35100b6590 0x7f35100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f3529600640 1 -- 172.21.15.39:0/826898191 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f351c107990 con 0x7f352c105d80 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.748+0000 7f3523400640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35100b6590 0x7f35100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.749+0000 7f353981a740 1 -- start start 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.749+0000 7f353981a740 1 Processor -- start 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.749+0000 7f3523400640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35100b6590 0x7f35100b8aa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35140099b0 tx=0x7f3514003e70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60ec18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 0x558e60ec1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ec2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558e60c7e810 con 0x558e60b102d0 2024-09-21T01:29:41.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558e60db8550 con 0x558e60ae7410 2024-09-21T01:29:41.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x558e60ec4a70 con 0x558e60aef5f0 2024-09-21T01:29:41.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f353981a740 10 client.0 _ll_register_callbacks cb 0x558e60dc8040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:29:41.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 0x558e60ec1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 0x558e60ec1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:35380/0 (socket says 172.21.15.39:35380) 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 -- 172.21.15.39:0/2645826101 learned_addr learned my addr 172.21.15.39:0/2645826101 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3530a00640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ec2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f352be00640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60ec18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 -- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 msgr2=0x558e60ec18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60ec18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 -- 172.21.15.39:0/2645826101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 msgr2=0x558e60ec2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ec2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.750+0000 7f3523400640 1 -- 172.21.15.39:0/2645826101 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e60daa200 con 0x558e60aef5f0 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f352be00640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60ec18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f3523400640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 0x558e60ec1e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f35140056c0 tx=0x7f3514011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=448e09ad8003e192 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f3519600640 10 client.0 ms_handle_connect on v2:172.21.15.160:3300/0 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f3519600640 1 -- 172.21.15.39:0/2645826101 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351402c050 con 0x558e60aef5f0 2024-09-21T01:29:41.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35140043d0 con 0x558e60aef5f0 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f3519600640 1 -- 172.21.15.39:0/2645826101 <== mon.1 v2:172.21.15.160:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f3514004bc0 con 0x558e60aef5f0 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f3519600640 1 -- 172.21.15.39:0/2645826101 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f351403f050 con 0x558e60aef5f0 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f353981a740 10 client.26605 fetch_fsmap learned FSMap version 68 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.751+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x558e60ed0780 con 0x558e60aef5f0 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f3519600640 1 -- 172.21.15.39:0/2645826101 <== mon.1 v2:172.21.15.160:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35140043d0 con 0x558e60aef5f0 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f3519600640 1 -- 172.21.15.39:0/2645826101 <== mon.1 v2:172.21.15.160:3300/0 5 ==== fsmap(e 68) ==== 8366+0+0 (secure 0 0 0) 0x7f3514018070 con 0x558e60aef5f0 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f353981a740 10 client.26605 fetch_fsmap finished waiting for FSMap version 68 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f353981a740 10 client.26605 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f353981a740 20 client.26605 populate_metadata read hostname 'smithi039' 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f353981a740 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60acec50 0x558e60ed3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f353981a740 4 client.26605 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e60e2ff40 con 0x558e60acec50 2024-09-21T01:29:41.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.752+0000 7f352be00640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60acec50 0x558e60ed3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.753+0000 7f352be00640 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60acec50 0x558e60ed3b40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae05d48b499ccec server_cookie=50f1ba419830b50f in_seq=0 out_seq=0 2024-09-21T01:29:41.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.753+0000 7f3519600640 10 client.26605 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f3519600640 1 -- 172.21.15.39:0/2645826101 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558e60e2ff40 con 0x558e60acec50 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f3519600640 10 client.26605 handle_command_reply: tid=0 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 client.26605 shutdown 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 msgr2=0x558e60ec1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 0x558e60ec1e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f35140056c0 tx=0x7f3514011810 comp rx=0 tx=0).stop 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 shutdown_connections 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60acec50 0x558e60ed3b40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ec2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60aef5f0 0x558e60ec1e00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 --2- 172.21.15.39:0/2645826101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60ae7410 0x558e60ec18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f3519600640 0 client.26605 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 >> 172.21.15.39:0/2645826101 conn(0x558e60b046c0 msgr2=0x558e60eb9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 shutdown_connections 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 1 -- 172.21.15.39:0/2645826101 wait complete. 2024-09-21T01:29:41.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.754+0000 7f353981a740 20 client.26605 trim_cache size 0 max 16384 2024-09-21T01:29:41.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.772+0000 7f353981a740 1 Processor -- start 2024-09-21T01:29:41.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.772+0000 7f353981a740 1 -- start start 2024-09-21T01:29:41.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f353981a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 0x558e60ec37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f353981a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ebead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f353981a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ebf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f353981a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f35080129c0 con 0x558e60b102d0 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f353981a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3508010200 con 0x558e60aef5f0 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f353981a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f350800e070 con 0x558e60ae7410 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f3530a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ebf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f352be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 0x558e60ec37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f352be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 0x558e60ec37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:35386/0 (socket says 172.21.15.39:35386) 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f352be00640 1 -- 172.21.15.39:0/4213535372 learned_addr learned my addr 172.21.15.39:0/4213535372 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:41.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.773+0000 7f3523400640 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ebead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f352be00640 1 -- 172.21.15.39:0/4213535372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 msgr2=0x558e60ebead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f352be00640 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ebead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f352be00640 1 -- 172.21.15.39:0/4213535372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 msgr2=0x558e60ebf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f352be00640 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ebf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f352be00640 1 -- 172.21.15.39:0/4213535372 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e60eb8580 con 0x558e60ae7410 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f352be00640 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 0x558e60ec37a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f351c0cd150 tx=0x7f351c009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f143bc056a1fc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f351be00640 1 -- 172.21.15.39:0/4213535372 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0d6dc0 con 0x558e60ae7410 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f351be00640 1 -- 172.21.15.39:0/4213535372 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f351c031d10 con 0x558e60ae7410 2024-09-21T01:29:41.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.774+0000 7f351be00640 1 -- 172.21.15.39:0/4213535372 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0d6b90 con 0x558e60ae7410 2024-09-21T01:29:41.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 -- 172.21.15.39:0/4213535372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 msgr2=0x558e60ec37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 0x558e60ec37a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f351c0cd150 tx=0x7f351c009630 comp rx=0 tx=0).stop 2024-09-21T01:29:41.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 -- 172.21.15.39:0/4213535372 shutdown_connections 2024-09-21T01:29:41.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60b102d0 0x558e60ebf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ebead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 --2- 172.21.15.39:0/4213535372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60ae7410 0x558e60ec37a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 -- 172.21.15.39:0/4213535372 >> 172.21.15.39:0/4213535372 conn(0x558e60b046c0 msgr2=0x558e60eb7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:41.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.775+0000 7f353981a740 1 -- 172.21.15.39:0/4213535372 shutdown_connections 2024-09-21T01:29:41.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.776+0000 7f353981a740 1 -- 172.21.15.39:0/4213535372 wait complete. 2024-09-21T01:29:41.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.776+0000 7f353981a740 1 -- start start 2024-09-21T01:29:41.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.776+0000 7f353981a740 1 Processor -- start 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60ae7410 0x558e60ec5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ec63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60ec6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f350800e6c0 con 0x558e60ae7410 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f350800fb30 con 0x558e60aef5f0 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f350800ed90 con 0x558e60b102d0 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f353981a740 10 client.0 _ll_register_callbacks cb 0x7f3514005370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f352be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60ae7410 0x558e60ec5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f3523400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ec63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f352be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60ae7410 0x558e60ec5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38342/0 (socket says 172.21.15.39:38342) 2024-09-21T01:29:41.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f352be00640 1 -- 172.21.15.39:0/2256987993 learned_addr learned my addr 172.21.15.39:0/2256987993 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:41.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.777+0000 7f3530a00640 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60ec6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3523400640 1 -- 172.21.15.39:0/2256987993 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 msgr2=0x558e60ec6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3523400640 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60ec6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3523400640 1 -- 172.21.15.39:0/2256987993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60ae7410 msgr2=0x558e60ec5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:29:41.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3523400640 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60ae7410 0x558e60ec5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3523400640 1 -- 172.21.15.39:0/2256987993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e60f45eb0 con 0x558e60aef5f0 2024-09-21T01:29:41.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3523400640 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ec63d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3514009380 tx=0x7f35140045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f56e087e428831 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:41.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.778+0000 7f3519600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-21T01:29:41.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f3519600640 1 -- 172.21.15.39:0/2256987993 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514047070 con 0x558e60aef5f0 2024-09-21T01:29:41.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f351c031d10 con 0x558e60aef5f0 2024-09-21T01:29:41.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f3519600640 1 -- 172.21.15.39:0/2256987993 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f3514004c90 con 0x558e60aef5f0 2024-09-21T01:29:41.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f3519600640 1 -- 172.21.15.39:0/2256987993 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f3514053050 con 0x558e60aef5f0 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f353981a740 10 client.26636 fetch_fsmap learned FSMap version 68 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x558e60f47110 con 0x558e60aef5f0 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.779+0000 7f3519600640 1 -- 172.21.15.39:0/2256987993 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f351c031d10 con 0x558e60aef5f0 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f3519600640 1 -- 172.21.15.39:0/2256987993 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 68) ==== 8366+0+0 (secure 0 0 0) 0x7f3514018040 con 0x558e60aef5f0 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f353981a740 10 client.26636 fetch_fsmap finished waiting for FSMap version 68 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f353981a740 10 client.26636 resolve_mds: validated gid 24764 aka daemon mds.b 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f353981a740 20 client.26636 populate_metadata read hostname 'smithi039' 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f353981a740 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60aa4000 0x558e60f498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f353981a740 4 client.26636 mds_command: new command op to 24764 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 --> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x558e60e2b9f0 con 0x558e60aa4000 2024-09-21T01:29:41.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.780+0000 7f352be00640 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60aa4000 0x558e60f498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:41.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f352be00640 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60aa4000 0x558e60f498b0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=552c7264dd641e29 server_cookie=f9762fc996a1bec4 in_seq=0 out_seq=0 2024-09-21T01:29:41.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f3519600640 10 client.26636 ms_handle_connect on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f3519600640 1 -- 172.21.15.39:0/2256987993 <== mds.0 v2:172.21.15.150:6838/103174700 1 ==== command_reply(tid 0: 0 ) ==== 8+0+81 (crc 0 0 0) 0x558e60e2b9f0 con 0x558e60aa4000 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f3519600640 10 client.26636 handle_command_reply: tid=0 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f353981a740 1 client.26636 shutdown 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 msgr2=0x558e60ec63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f353981a740 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ec63d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3514009380 tx=0x7f35140045e0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 shutdown_connections 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.781+0000 7f353981a740 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:6838/103174700,v1:172.21.15.150:6839/103174700] conn(0x558e60aa4000 0x558e60f498b0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x558e60b102d0 0x558e60ec6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558e60aef5f0 0x558e60ec63d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 1 --2- 172.21.15.39:0/2256987993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558e60ae7410 0x558e60ec5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 >> 172.21.15.39:0/2256987993 conn(0x558e60b046c0 msgr2=0x558e60eb7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f3519600640 0 client.26636 ms_handle_reset on v2:172.21.15.150:6838/103174700 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 shutdown_connections 2024-09-21T01:29:41.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 1 -- 172.21.15.39:0/2256987993 wait complete. 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.782+0000 7f353981a740 20 client.26636 trim_cache size 0 max 16384 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 68, 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "leader": 24764, 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "set_version": 100, 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: "sets": {} 2024-09-21T01:29:41.785 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35100b6590 msgr2=0x7f35100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35100b6590 0x7f35100b8aa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35140099b0 tx=0x7f3514003e70 comp rx=0 tx=0).stop 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 msgr2=0x7f352c19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 0x7f352c19c6f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f351c005ab0 tx=0x7f351c003c10 comp rx=0 tx=0).stop 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 shutdown_connections 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352c10c010 0x7f352c1a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f352c10b6c0 0x7f352c19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f35100b6590 0x7f35100b8aa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 --2- 172.21.15.39:0/826898191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f352c105d80 0x7f352c19c6f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 >> 172.21.15.39:0/826898191 conn(0x7f352c101810 msgr2=0x7f352c103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:41.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 shutdown_connections 2024-09-21T01:29:41.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:41.783+0000 7f3532a00640 1 -- 172.21.15.39:0/826898191 wait complete. 2024-09-21T01:29:42.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:41 smithi039 ceph-mon[30415]: pgmap v1214: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 7.1 MiB/s wr, 342 op/s 2024-09-21T01:29:42.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:41 smithi150 ceph-mon[34945]: pgmap v1214: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 7.1 MiB/s wr, 342 op/s 2024-09-21T01:29:42.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:41 smithi160 ceph-mon[34696]: pgmap v1214: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 7.1 MiB/s wr, 342 op/s 2024-09-21T01:29:42.529 INFO:tasks.quiescer:done joining Quiescers 2024-09-21T01:29:42.530 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-09-21T01:29:42.539 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-09-21T01:29:42.540 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T01:29:42.540 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:29:42.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-09-21T01:29:42.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-09-21T01:29:42.677 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-21T01:29:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 -- 172.21.15.39:0/2075416912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109430 msgr2=0x7f94ac109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 --2- 172.21.15.39:0/2075416912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109430 0x7f94ac109840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9494009900 tx=0x7f949402a410 comp rx=0 tx=0).stop 2024-09-21T01:29:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 -- 172.21.15.39:0/2075416912 shutdown_connections 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 --2- 172.21.15.39:0/2075416912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109d80 0x7f94ac1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 --2- 172.21.15.39:0/2075416912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109430 0x7f94ac109840 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 --2- 172.21.15.39:0/2075416912 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 0x7f94ac103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 -- 172.21.15.39:0/2075416912 >> 172.21.15.39:0/2075416912 conn(0x7f94ac0ff500 msgr2=0x7f94ac101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 -- 172.21.15.39:0/2075416912 shutdown_connections 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.053+0000 7f94b1000640 1 -- 172.21.15.39:0/2075416912 wait complete. 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.054+0000 7f94b1000640 1 Processor -- start 2024-09-21T01:29:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 -- start start 2024-09-21T01:29:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 0x7f94ac19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109430 0x7f94ac19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109d80 0x7f94ac19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac113a90 con 0x7f94ac109d80 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac113910 con 0x7f94ac109430 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94b1000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac113c10 con 0x7f94ac103b00 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94a3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109430 0x7f94ac19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94aa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109d80 0x7f94ac19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94a9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 0x7f94ac19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94a9e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 0x7f94ac19a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53732/0 (socket says 172.21.15.39:53732) 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94a3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109430 0x7f94ac19aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50436/0 (socket says 172.21.15.39:50436) 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.055+0000 7f94a9e00640 1 -- 172.21.15.39:0/1937688265 learned_addr learned my addr 172.21.15.39:0/1937688265 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94aa800640 1 -- 172.21.15.39:0/1937688265 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 msgr2=0x7f94ac19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94aa800640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 0x7f94ac19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94aa800640 1 -- 172.21.15.39:0/1937688265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109430 msgr2=0x7f94ac19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94aa800640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109430 0x7f94ac19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94aa800640 1 -- 172.21.15.39:0/1937688265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ac1a03e0 con 0x7f94ac109d80 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94aa800640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109d80 0x7f94ac19fce0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009fd0 tx=0x7f94a400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a400ce20 con 0x7f94ac109d80 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94ac1a06d0 con 0x7f94ac109d80 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94a4005020 con 0x7f94ac109d80 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94ac1a6e60 con 0x7f94ac109d80 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.056+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4018670 con 0x7f94ac109d80 2024-09-21T01:29:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.058+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94ac103f10 con 0x7f94ac109d80 2024-09-21T01:29:43.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.060+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f94a40054d0 con 0x7f94ac109d80 2024-09-21T01:29:43.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.061+0000 7f94a2a00640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94900b6680 0x7f94900b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.061+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f94a40e8320 con 0x7f94ac109d80 2024-09-21T01:29:43.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.062+0000 7f94a9e00640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94900b6680 0x7f94900b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.062+0000 7f94a9e00640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94900b6680 0x7f94900b8b90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f949c009cd0 tx=0x7f949c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:43.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.063+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f94a4055710 con 0x7f94ac109d80 2024-09-21T01:29:43.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.280+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f94ac063800 con 0x7f94900b6680 2024-09-21T01:29:43.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.294+0000 7f94a2a00640 1 -- 172.21.15.39:0/1937688265 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f94ac063800 con 0x7f94900b6680 2024-09-21T01:29:43.296 INFO:teuthology.orchestra.run.smithi039.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-21T00:00:00", "created": "2024-09-21T01:03:45", "first": "2024-09-21T01:04:00", "last": "2024-09-21T01:29:00", "last_pruned": "2024-09-21T01:29:00", "created_count": 26, "pruned_count": 20, "active": true} 2024-09-21T01:29:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94900b6680 msgr2=0x7f94900b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94900b6680 0x7f94900b8b90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f949c009cd0 tx=0x7f949c005f90 comp rx=0 tx=0).stop 2024-09-21T01:29:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109d80 msgr2=0x7f94ac19fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109d80 0x7f94ac19fce0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009fd0 tx=0x7f94a400ef10 comp rx=0 tx=0).stop 2024-09-21T01:29:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 shutdown_connections 2024-09-21T01:29:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac109d80 0x7f94ac19fce0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f94900b6680 0x7f94900b8b90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac109430 0x7f94ac19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 --2- 172.21.15.39:0/1937688265 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f94ac103b00 0x7f94ac19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 >> 172.21.15.39:0/1937688265 conn(0x7f94ac0ff500 msgr2=0x7f94ac106a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 shutdown_connections 2024-09-21T01:29:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.297+0000 7f94b1000640 1 -- 172.21.15.39:0/1937688265 wait complete. 2024-09-21T01:29:43.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 -- 172.21.15.39:0/1148472168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15441141d0 msgr2=0x7f15441166a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 --2- 172.21.15.39:0/1148472168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15441141d0 0x7f15441166a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f153c009ca0 tx=0x7f153c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 -- 172.21.15.39:0/1148472168 shutdown_connections 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 --2- 172.21.15.39:0/1148472168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15441141d0 0x7f15441166a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 --2- 172.21.15.39:0/1148472168 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1544075dd0 0x7f15440761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 --2- 172.21.15.39:0/1148472168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544077390 0x7f15440757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 -- 172.21.15.39:0/1148472168 >> 172.21.15.39:0/1148472168 conn(0x7f15440ff330 msgr2=0x7f1544101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 -- 172.21.15.39:0/1148472168 shutdown_connections 2024-09-21T01:29:43.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.549+0000 7f1549600640 1 -- 172.21.15.39:0/1148472168 wait complete. 2024-09-21T01:29:43.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.550+0000 7f1549600640 1 Processor -- start 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 -- start start 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 0x7f15441a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 0x7f15441a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15441141d0 0x7f15441a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1544119e60 con 0x7f1544077390 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1544119ce0 con 0x7f1544075dd0 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1549600640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1544119fe0 con 0x7f15441141d0 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f153be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 0x7f15441a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1543e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 0x7f15441a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f153be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 0x7f15441a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47148/0 (socket says 172.21.15.39:47148) 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f153be00640 1 -- 172.21.15.39:0/1622728247 learned_addr learned my addr 172.21.15.39:0/1622728247 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1543e00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 0x7f15441a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50452/0 (socket says 172.21.15.39:50452) 2024-09-21T01:29:43.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.551+0000 7f1548c00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15441141d0 0x7f15441a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f1543e00640 1 -- 172.21.15.39:0/1622728247 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15441141d0 msgr2=0x7f15441a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f1543e00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15441141d0 0x7f15441a5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f1543e00640 1 -- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 msgr2=0x7f15441a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f1543e00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 0x7f15441a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f1543e00640 1 -- 172.21.15.39:0/1622728247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15441a66f0 con 0x7f1544075dd0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f153be00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 0x7f15441a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.552+0000 7f1543e00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 0x7f15441a07c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f153400e9f0 tx=0x7f153400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.553+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153400ccc0 con 0x7f1544075dd0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.553+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15441a69e0 con 0x7f1544075dd0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.553+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15340108e0 con 0x7f1544075dd0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.553+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1534018880 con 0x7f1544075dd0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.553+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15441ad170 con 0x7f1544075dd0 2024-09-21T01:29:43.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.554+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15440761e0 con 0x7f1544075dd0 2024-09-21T01:29:43.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.556+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f1534010430 con 0x7f1544075dd0 2024-09-21T01:29:43.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.558+0000 7f1541600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15280b6680 0x7f15280b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:43.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.558+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f15340e82f0 con 0x7f1544075dd0 2024-09-21T01:29:43.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.558+0000 7f153be00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15280b6680 0x7f15280b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:43.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.559+0000 7f153be00640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15280b6680 0x7f15280b8b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1544115f00 tx=0x7f152c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:43.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.560+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f15340b27e0 con 0x7f1544075dd0 2024-09-21T01:29:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.784+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 --> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1544063800 con 0x7f15280b6680 2024-09-21T01:29:43.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.797+0000 7f1541600640 1 -- 172.21.15.39:0/1622728247 <== mgr.14223 v2:172.21.15.39:6800/2923702906 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1544063800 con 0x7f15280b6680 2024-09-21T01:29:43.799 INFO:teuthology.orchestra.run.smithi039.stdout:/ 1m 6m3h 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15280b6680 msgr2=0x7f15280b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15280b6680 0x7f15280b8b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1544115f00 tx=0x7f152c005f90 comp rx=0 tx=0).stop 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 msgr2=0x7f15441a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 0x7f15441a07c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f153400e9f0 tx=0x7f153400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 shutdown_connections 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15441141d0 0x7f15441a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1544077390 0x7f15441a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f15280b6680 0x7f15280b8b90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 --2- 172.21.15.39:0/1622728247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544075dd0 0x7f15441a07c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:43.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 >> 172.21.15.39:0/1622728247 conn(0x7f15440ff330 msgr2=0x7f15441100d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:43.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 shutdown_connections 2024-09-21T01:29:43.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:43.800+0000 7f1549600640 1 -- 172.21.15.39:0/1622728247 wait complete. 2024-09-21T01:29:43.826 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-21T01:29:43.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T01:29:44.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 -- 172.21.15.39:0/1358330109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80076100 msgr2=0x7f8b8010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 --2- 172.21.15.39:0/1358330109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80076100 0x7f8b8010f900 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c009900 tx=0x7f8b6c02a3f0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 -- 172.21.15.39:0/1358330109 shutdown_connections 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 --2- 172.21.15.39:0/1358330109 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b8010fe40 0x7f8b801122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 --2- 172.21.15.39:0/1358330109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80076100 0x7f8b8010f900 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 --2- 172.21.15.39:0/1358330109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b800757b0 0x7f8b80075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 -- 172.21.15.39:0/1358330109 >> 172.21.15.39:0/1358330109 conn(0x7f8b800ff350 msgr2=0x7f8b801017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 -- 172.21.15.39:0/1358330109 shutdown_connections 2024-09-21T01:29:44.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.067+0000 7f8b86a00640 1 -- 172.21.15.39:0/1358330109 wait complete. 2024-09-21T01:29:44.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.068+0000 7f8b86a00640 1 Processor -- start 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.068+0000 7f8b86a00640 1 -- start start 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b800757b0 0x7f8b8019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86a00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 0x7f8b8019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b8010fe40 0x7f8b801a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8b80115a60 con 0x7f8b8010fe40 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801158e0 con 0x7f8b800757b0 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86a00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8b80115be0 con 0x7f8b80076100 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b84c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 0x7f8b8019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b84c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 0x7f8b8019c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:53772/0 (socket says 172.21.15.39:53772) 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b84c00640 1 -- 172.21.15.39:0/144738223 learned_addr learned my addr 172.21.15.39:0/144738223 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b85600640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b800757b0 0x7f8b8019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b86000640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b8010fe40 0x7f8b801a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b85600640 1 -- 172.21.15.39:0/144738223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 msgr2=0x7f8b8019c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b85600640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 0x7f8b8019c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b85600640 1 -- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b8010fe40 msgr2=0x7f8b801a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b85600640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b8010fe40 0x7f8b801a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b85600640 1 -- 172.21.15.39:0/144738223 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b801a2350 con 0x7f8b800757b0 2024-09-21T01:29:44.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.069+0000 7f8b84c00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 0x7f8b8019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:44.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.070+0000 7f8b85600640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b800757b0 0x7f8b8019c420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400e9f0 tx=0x7f8b7400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:44.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.070+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400ccc0 con 0x7f8b800757b0 2024-09-21T01:29:44.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.070+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b801a2640 con 0x7f8b800757b0 2024-09-21T01:29:44.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.070+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b740108e0 con 0x7f8b800757b0 2024-09-21T01:29:44.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.070+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74018880 con 0x7f8b800757b0 2024-09-21T01:29:44.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.070+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b801a8dd0 con 0x7f8b800757b0 2024-09-21T01:29:44.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.071+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b80075bc0 con 0x7f8b800757b0 2024-09-21T01:29:44.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.073+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8b74010430 con 0x7f8b800757b0 2024-09-21T01:29:44.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.075+0000 7f8b7e000640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8b680b6680 0x7f8b680b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.075+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f8b740e82f0 con 0x7f8b800757b0 2024-09-21T01:29:44.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.075+0000 7f8b84c00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8b680b6680 0x7f8b680b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.076+0000 7f8b84c00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8b680b6680 0x7f8b680b8b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c006010 tx=0x7f8b6c005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:44.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.077+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8b740b27e0 con 0x7f8b800757b0 2024-09-21T01:29:44.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:43 smithi039 ceph-mon[30415]: pgmap v1215: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 6.2 MiB/s wr, 300 op/s 2024-09-21T01:29:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.276+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b801a9140 con 0x7f8b800757b0 2024-09-21T01:29:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.279+0000 7f8b7e000640 1 -- 172.21.15.39:0/144738223 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+25519 (secure 0 0 0) 0x7f8b740b7720 con 0x7f8b800757b0 2024-09-21T01:29:44.280 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:29:44.280 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":130,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:29:14.755759+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":129,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":129,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:29:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8b680b6680 msgr2=0x7f8b680b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8b680b6680 0x7f8b680b8b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c006010 tx=0x7f8b6c005a20 comp rx=0 tx=0).stop 2024-09-21T01:29:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b800757b0 msgr2=0x7f8b8019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b800757b0 0x7f8b8019c420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400e9f0 tx=0x7f8b7400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 shutdown_connections 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b8010fe40 0x7f8b801a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8b80076100 0x7f8b8019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f8b680b6680 0x7f8b680b8b90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 --2- 172.21.15.39:0/144738223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b800757b0 0x7f8b8019c420 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 >> 172.21.15.39:0/144738223 conn(0x7f8b800ff350 msgr2=0x7f8b80109950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 shutdown_connections 2024-09-21T01:29:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.281+0000 7f8b86a00640 1 -- 172.21.15.39:0/144738223 wait complete. 2024-09-21T01:29:44.305 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T00:55:22.265381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-21T01:00:37.542995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-21T01:00:40.673682+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 129, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-21T01:00:50.564539+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 129, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-21T01:29:44.306 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-21T01:29:44.313 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:43 smithi150 ceph-mon[34945]: pgmap v1215: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 6.2 MiB/s wr, 300 op/s 2024-09-21T01:29:44.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:43 smithi160 ceph-mon[34696]: pgmap v1215: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 6.2 MiB/s wr, 300 op/s 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 -- 172.21.15.39:0/1277253687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 msgr2=0x7f06e4075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 --2- 172.21.15.39:0/1277253687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e4075b40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009900 tx=0x7f06d402a410 comp rx=0 tx=0).stop 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 -- 172.21.15.39:0/1277253687 shutdown_connections 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 --2- 172.21.15.39:0/1277253687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e410fe80 0x7f06e41122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 --2- 172.21.15.39:0/1277253687 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e4076080 0x7f06e410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 --2- 172.21.15.39:0/1277253687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e4075b40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.532+0000 7f06e9200640 1 -- 172.21.15.39:0/1277253687 >> 172.21.15.39:0/1277253687 conn(0x7f06e40ff2d0 msgr2=0x7f06e4101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.533+0000 7f06e9200640 1 -- 172.21.15.39:0/1277253687 shutdown_connections 2024-09-21T01:29:44.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.533+0000 7f06e9200640 1 -- 172.21.15.39:0/1277253687 wait complete. 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 Processor -- start 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 -- start start 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e4076080 0x7f06e419c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e410fe80 0x7f06e41a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41159c0 con 0x7f06e4075730 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06e4115840 con 0x7f06e4076080 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e9200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f06e4115b40 con 0x7f06e410fe80 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e3e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e410fe80 0x7f06e41a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47202/0 (socket says 172.21.15.39:47202) 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.534+0000 7f06e3400640 1 -- 172.21.15.39:0/4155002478 learned_addr learned my addr 172.21.15.39:0/4155002478 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e2a00640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e4076080 0x7f06e419c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3e00640 1 -- 172.21.15.39:0/4155002478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e4076080 msgr2=0x7f06e419c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3e00640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e4076080 0x7f06e419c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3e00640 1 -- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 msgr2=0x7f06e419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3e00640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3e00640 1 -- 172.21.15.39:0/4155002478 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e41a2370 con 0x7f06e410fe80 2024-09-21T01:29:44.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3400640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06e3e00640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e410fe80 0x7f06e41a1c70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009fd0 tx=0x7f06dc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.535+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc00ce20 con 0x7f06e410fe80 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.536+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06dc005020 con 0x7f06e410fe80 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.536+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc010690 con 0x7f06e410fe80 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.536+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e41a2660 con 0x7f06e410fe80 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.536+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e41a8df0 con 0x7f06e410fe80 2024-09-21T01:29:44.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.537+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06e4075b40 con 0x7f06e410fe80 2024-09-21T01:29:44.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.540+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f06dc0054d0 con 0x7f06e410fe80 2024-09-21T01:29:44.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.541+0000 7f06dbe00640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06c80b6700 0x7f06c80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:44.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.541+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f06dc0e8170 con 0x7f06e410fe80 2024-09-21T01:29:44.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.541+0000 7f06e3400640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06c80b6700 0x7f06c80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:44.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.542+0000 7f06e3400640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06c80b6700 0x7f06c80b8c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f06d4031440 tx=0x7f06d40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:44.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.543+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f06dc0ae280 con 0x7f06e410fe80 2024-09-21T01:29:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.747+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f06e41a9120 con 0x7f06e410fe80 2024-09-21T01:29:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.748+0000 7f06dbe00640 1 -- 172.21.15.39:0/4155002478 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) ==== 93+0+10 (secure 0 0 0) 0x7f06dc0b2660 con 0x7f06e410fe80 2024-09-21T01:29:44.749 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-09-21T01:29:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06c80b6700 msgr2=0x7f06c80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06c80b6700 0x7f06c80b8c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f06d4031440 tx=0x7f06d40058d0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e410fe80 msgr2=0x7f06e41a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e410fe80 0x7f06e41a1c70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009fd0 tx=0x7f06dc00ef10 comp rx=0 tx=0).stop 2024-09-21T01:29:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 shutdown_connections 2024-09-21T01:29:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06e410fe80 0x7f06e41a1c70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f06c80b6700 0x7f06c80b8c10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e4076080 0x7f06e419c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 --2- 172.21.15.39:0/4155002478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06e4075730 0x7f06e419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 >> 172.21.15.39:0/4155002478 conn(0x7f06e40ff2d0 msgr2=0x7f06e4101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 shutdown_connections 2024-09-21T01:29:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:44.751+0000 7f06e9200640 1 -- 172.21.15.39:0/4155002478 wait complete. 2024-09-21T01:29:44.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-21T01:29:45.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 -- 172.21.15.39:0/4201800436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 msgr2=0x7f1ca0109630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/4201800436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca0109630 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009900 tx=0x7f1c8c02a410 comp rx=0 tx=0).stop 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 -- 172.21.15.39:0/4201800436 shutdown_connections 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/4201800436 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 0x7f1ca01103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/4201800436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca0109630 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/4201800436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca01038f0 0x7f1ca0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 -- 172.21.15.39:0/4201800436 >> 172.21.15.39:0/4201800436 conn(0x7f1ca00ff330 msgr2=0x7f1ca0101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 -- 172.21.15.39:0/4201800436 shutdown_connections 2024-09-21T01:29:45.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.004+0000 7f1ca6e00640 1 -- 172.21.15.39:0/4201800436 wait complete. 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 Processor -- start 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 -- start start 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca01038f0 0x7f1ca019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 0x7f1ca019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca0113840 con 0x7f1ca0109220 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01136c0 con 0x7f1ca01038f0 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.005+0000 7f1ca6e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01139c0 con 0x7f1ca0109b70 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 0x7f1ca019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47212/0 (socket says 172.21.15.39:47212) 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca5a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca01038f0 0x7f1ca019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca5000640 1 -- 172.21.15.39:0/2096363068 learned_addr learned my addr 172.21.15.39:0/2096363068 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 -- 172.21.15.39:0/2096363068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca01038f0 msgr2=0x7f1ca019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca01038f0 0x7f1ca019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 -- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 msgr2=0x7f1ca019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 -- 172.21.15.39:0/2096363068 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca01a0190 con 0x7f1ca0109b70 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca5000640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.006+0000 7f1ca6400640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 0x7f1ca019fa90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800e940 tx=0x7f1c9800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.007+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98004d20 con 0x7f1ca0109b70 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.007+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca01a0480 con 0x7f1ca0109b70 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.007+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c98005020 con 0x7f1ca0109b70 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.007+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c980116d0 con 0x7f1ca0109b70 2024-09-21T01:29:45.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.007+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca01a6c10 con 0x7f1ca0109b70 2024-09-21T01:29:45.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.008+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca0104b60 con 0x7f1ca0109b70 2024-09-21T01:29:45.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.011+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f1c98005460 con 0x7f1ca0109b70 2024-09-21T01:29:45.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.012+0000 7f1c9e000640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1c880b6680 0x7f1c880b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.012+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f1c980e7ec0 con 0x7f1ca0109b70 2024-09-21T01:29:45.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.012+0000 7f1ca5a00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1c880b6680 0x7f1c880b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.013+0000 7f1ca5a00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1c880b6680 0x7f1c880b8b90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005d00 tx=0x7f1c9400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:45.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.014+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1c98016030 con 0x7f1ca0109b70 2024-09-21T01:29:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:44 smithi039 ceph-mon[30415]: from='client.16605 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:44 smithi039 ceph-mon[30415]: from='client.26648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:45.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:44 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/144738223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:29:45.154 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:44 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/4155002478' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T01:29:45.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.212+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ca019b380 con 0x7f1ca0109b70 2024-09-21T01:29:45.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.213+0000 7f1c9e000640 1 -- 172.21.15.39:0/2096363068 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) ==== 104+0+11 (secure 0 0 0) 0x7f1ca019b380 con 0x7f1ca0109b70 2024-09-21T01:29:45.214 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-09-21T01:29:45.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1c880b6680 msgr2=0x7f1c880b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1c880b6680 0x7f1c880b8b90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005d00 tx=0x7f1c9400b410 comp rx=0 tx=0).stop 2024-09-21T01:29:45.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 msgr2=0x7f1ca019fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 0x7f1ca019fa90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800e940 tx=0x7f1c9800ee00 comp rx=0 tx=0).stop 2024-09-21T01:29:45.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 shutdown_connections 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1ca0109b70 0x7f1ca019fa90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca0109220 0x7f1ca019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f1c880b6680 0x7f1c880b8b90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 --2- 172.21.15.39:0/2096363068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca01038f0 0x7f1ca019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 >> 172.21.15.39:0/2096363068 conn(0x7f1ca00ff330 msgr2=0x7f1ca0101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 shutdown_connections 2024-09-21T01:29:45.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.215+0000 7f1ca6e00640 1 -- 172.21.15.39:0/2096363068 wait complete. 2024-09-21T01:29:45.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-21T01:29:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:44 smithi150 ceph-mon[34945]: from='client.16605 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:44 smithi150 ceph-mon[34945]: from='client.26648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:44 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/144738223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:29:45.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:44 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/4155002478' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T01:29:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:44 smithi160 ceph-mon[34696]: from='client.16605 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:44 smithi160 ceph-mon[34696]: from='client.26648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:44 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/144738223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:29:45.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:44 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/4155002478' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T01:29:45.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.468+0000 7fb2e1200640 1 -- 172.21.15.39:0/4050622785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc105060 msgr2=0x7fb2dc10d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.468+0000 7fb2e1200640 1 --2- 172.21.15.39:0/4050622785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc105060 0x7fb2dc10d5a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009900 tx=0x7fb2c802a410 comp rx=0 tx=0).stop 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.468+0000 7fb2e1200640 1 -- 172.21.15.39:0/4050622785 shutdown_connections 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.469+0000 7fb2e1200640 1 --2- 172.21.15.39:0/4050622785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc10db70 0x7fb2dc110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.469+0000 7fb2e1200640 1 --2- 172.21.15.39:0/4050622785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc105060 0x7fb2dc10d5a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.469+0000 7fb2e1200640 1 --2- 172.21.15.39:0/4050622785 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2dc104710 0x7fb2dc104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.469+0000 7fb2e1200640 1 -- 172.21.15.39:0/4050622785 >> 172.21.15.39:0/4050622785 conn(0x7fb2dc0fe1e0 msgr2=0x7fb2dc100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.469+0000 7fb2e1200640 1 -- 172.21.15.39:0/4050622785 shutdown_connections 2024-09-21T01:29:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.469+0000 7fb2e1200640 1 -- 172.21.15.39:0/4050622785 wait complete. 2024-09-21T01:29:45.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 Processor -- start 2024-09-21T01:29:45.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 -- start start 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 0x7fb2dc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2dc105060 0x7fb2dc19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 0x7fb2dc19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1137b0 con 0x7fb2dc10db70 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc113630 con 0x7fb2dc104710 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.470+0000 7fb2e1200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc113930 con 0x7fb2dc105060 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 0x7fb2dc19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2db400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 0x7fb2dc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2dc105060 0x7fb2dc19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 0x7fb2dc19faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47232/0 (socket says 172.21.15.39:47232) 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2db400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 0x7fb2dc19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:50544/0 (socket says 172.21.15.39:50544) 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 -- 172.21.15.39:0/1133971258 learned_addr learned my addr 172.21.15.39:0/1133971258 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 -- 172.21.15.39:0/1133971258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2dc105060 msgr2=0x7fb2dc19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2dc105060 0x7fb2dc19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 -- 172.21.15.39:0/1133971258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 msgr2=0x7fb2dc19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 0x7fb2dc19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2dbe00640 1 -- 172.21.15.39:0/1133971258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc1a01a0 con 0x7fb2dc10db70 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.471+0000 7fb2db400640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 0x7fb2dc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.472+0000 7fb2dbe00640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 0x7fb2dc19faa0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400eb10 tx=0x7fb2d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.472+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4004b20 con 0x7fb2dc10db70 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.472+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d4004cc0 con 0x7fb2dc10db70 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.472+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4010670 con 0x7fb2dc10db70 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.472+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2dc1a0490 con 0x7fb2dc10db70 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.472+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2dc1a6c20 con 0x7fb2dc10db70 2024-09-21T01:29:45.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.473+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2dc104b20 con 0x7fb2dc10db70 2024-09-21T01:29:45.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.475+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fb2d4005000 con 0x7fb2dc10db70 2024-09-21T01:29:45.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.477+0000 7fb2d3e00640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2c00b6700 0x7fb2c00b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.477+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7fb2d40e82e0 con 0x7fb2dc10db70 2024-09-21T01:29:45.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.477+0000 7fb2db400640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2c00b6700 0x7fb2c00b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.478+0000 7fb2db400640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2c00b6700 0x7fb2c00b8c10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009fc0 tx=0x7fb2cc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:45.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.479+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb2d4056710 con 0x7fb2dc10db70 2024-09-21T01:29:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.675+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2dc1a6ec0 con 0x7fb2dc10db70 2024-09-21T01:29:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.675+0000 7fb2d3e00640 1 -- 172.21.15.39:0/1133971258 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) ==== 100+0+11 (secure 0 0 0) 0x7fb2d40ae3f0 con 0x7fb2dc10db70 2024-09-21T01:29:45.676 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2c00b6700 msgr2=0x7fb2c00b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2c00b6700 0x7fb2c00b8c10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009fc0 tx=0x7fb2cc005f90 comp rx=0 tx=0).stop 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 msgr2=0x7fb2dc19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 0x7fb2dc19faa0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400eb10 tx=0x7fb2d400c2d0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 shutdown_connections 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2dc10db70 0x7fb2dc19faa0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2dc105060 0x7fb2dc19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb2c00b6700 0x7fb2c00b8c10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 --2- 172.21.15.39:0/1133971258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2dc104710 0x7fb2dc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 >> 172.21.15.39:0/1133971258 conn(0x7fb2dc0fe1e0 msgr2=0x7fb2dc1076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:45.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 shutdown_connections 2024-09-21T01:29:45.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.678+0000 7fb2e1200640 1 -- 172.21.15.39:0/1133971258 wait complete. 2024-09-21T01:29:45.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-21T01:29:45.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 -- 172.21.15.39:0/3580768847 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff194105ae0 msgr2=0x7ff194105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 --2- 172.21.15.39:0/3580768847 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff194105ae0 0x7ff194105ef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff184009900 tx=0x7ff18402a400 comp rx=0 tx=0).stop 2024-09-21T01:29:45.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 -- 172.21.15.39:0/3580768847 shutdown_connections 2024-09-21T01:29:45.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 --2- 172.21.15.39:0/3580768847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff19410bd60 0x7ff194112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 --2- 172.21.15.39:0/3580768847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410b410 0x7ff19410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 --2- 172.21.15.39:0/3580768847 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff194105ae0 0x7ff194105ef0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 -- 172.21.15.39:0/3580768847 >> 172.21.15.39:0/3580768847 conn(0x7ff194101560 msgr2=0x7ff1941039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:45.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 -- 172.21.15.39:0/3580768847 shutdown_connections 2024-09-21T01:29:45.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.933+0000 7ff198e00640 1 -- 172.21.15.39:0/3580768847 wait complete. 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 Processor -- start 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 -- start start 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff194105ae0 0x7ff19419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff19410b410 0x7ff19419c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 0x7ff1941a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff194115a70 con 0x7ff19410bd60 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941158f0 con 0x7ff194105ae0 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.934+0000 7ff198e00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff194115bf0 con 0x7ff19410b410 2024-09-21T01:29:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff193e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 0x7ff1941a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff18be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff19410b410 0x7ff19419c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff193e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 0x7ff1941a1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47246/0 (socket says 172.21.15.39:47246) 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff193e00640 1 -- 172.21.15.39:0/40306426 learned_addr learned my addr 172.21.15.39:0/40306426 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff193400640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff194105ae0 0x7ff19419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff18be00640 1 -- 172.21.15.39:0/40306426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff194105ae0 msgr2=0x7ff19419c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff18be00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff194105ae0 0x7ff19419c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff18be00640 1 -- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 msgr2=0x7ff1941a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff18be00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 0x7ff1941a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff18be00640 1 -- 172.21.15.39:0/40306426 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1941a2380 con 0x7ff19410b410 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.935+0000 7ff193e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 0x7ff1941a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.936+0000 7ff18be00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff19410b410 0x7ff19419c9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff19406c640 tx=0x7ff17c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.936+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00cc40 con 0x7ff19410b410 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.936+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1941a2670 con 0x7ff19410b410 2024-09-21T01:29:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.936+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff17c00cde0 con 0x7ff19410b410 2024-09-21T01:29:45.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.936+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c018760 con 0x7ff19410b410 2024-09-21T01:29:45.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.936+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1941a8e00 con 0x7ff19410b410 2024-09-21T01:29:45.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.937+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff194105ef0 con 0x7ff19410b410 2024-09-21T01:29:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.939+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff17c010a90 con 0x7ff19410b410 2024-09-21T01:29:45.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.941+0000 7ff190c00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1780b6680 0x7ff1780b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:45.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.941+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7ff17c0e8270 con 0x7ff19410b410 2024-09-21T01:29:45.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.941+0000 7ff193400640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1780b6680 0x7ff1780b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:45.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.942+0000 7ff193400640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1780b6680 0x7ff1780b8b90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff184005ab0 tx=0x7ff184005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:45.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:45.943+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff17c0b27c0 con 0x7ff19410b410 2024-09-21T01:29:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.140+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff19410db90 con 0x7ff19410b410 2024-09-21T01:29:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.141+0000 7ff190c00640 1 -- 172.21.15.39:0/40306426 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v130) ==== 103+0+11 (secure 0 0 0) 0x7ff17c0b7700 con 0x7ff19410b410 2024-09-21T01:29:46.142 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.143+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1780b6680 msgr2=0x7ff1780b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.143+0000 7ff198e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1780b6680 0x7ff1780b8b90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff184005ab0 tx=0x7ff184005a20 comp rx=0 tx=0).stop 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.143+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff19410b410 msgr2=0x7ff19419c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.143+0000 7ff198e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff19410b410 0x7ff19419c9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff19406c640 tx=0x7ff17c00ee50 comp rx=0 tx=0).stop 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 shutdown_connections 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19410bd60 0x7ff1941a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff19410b410 0x7ff19419c9f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff1780b6680 0x7ff1780b8b90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 --2- 172.21.15.39:0/40306426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff194105ae0 0x7ff19419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 >> 172.21.15.39:0/40306426 conn(0x7ff194101560 msgr2=0x7ff19410aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 shutdown_connections 2024-09-21T01:29:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.144+0000 7ff198e00640 1 -- 172.21.15.39:0/40306426 wait complete. 2024-09-21T01:29:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:45 smithi039 ceph-mon[30415]: pgmap v1216: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.8 MiB/s wr, 209 op/s 2024-09-21T01:29:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2096363068' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T01:29:46.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:45 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1133971258' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T01:29:46.167 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-21T01:29:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:45 smithi150 ceph-mon[34945]: pgmap v1216: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.8 MiB/s wr, 209 op/s 2024-09-21T01:29:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2096363068' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T01:29:46.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:45 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1133971258' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T01:29:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:45 smithi160 ceph-mon[34696]: pgmap v1216: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.8 MiB/s wr, 209 op/s 2024-09-21T01:29:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2096363068' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T01:29:46.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:45 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1133971258' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T01:29:46.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 -- 172.21.15.39:0/1102923310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 msgr2=0x7f25b410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:46.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 --2- 172.21.15.39:0/1102923310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b410b820 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009900 tx=0x7f25a002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 -- 172.21.15.39:0/1102923310 shutdown_connections 2024-09-21T01:29:46.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 --2- 172.21.15.39:0/1102923310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b410bd60 0x7f25b4112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 --2- 172.21.15.39:0/1102923310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b410b820 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 --2- 172.21.15.39:0/1102923310 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f25b4105ae0 0x7f25b4105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 -- 172.21.15.39:0/1102923310 >> 172.21.15.39:0/1102923310 conn(0x7f25b4101560 msgr2=0x7f25b41039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 -- 172.21.15.39:0/1102923310 shutdown_connections 2024-09-21T01:29:46.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.392+0000 7f25ba800640 1 -- 172.21.15.39:0/1102923310 wait complete. 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 Processor -- start 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 -- start start 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f25b4105ae0 0x7f25b419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b410bd60 0x7f25b41a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f25b4115a70 con 0x7f25b410b410 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f25b41158f0 con 0x7f25b410bd60 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.393+0000 7f25ba800640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f25b4115bf0 con 0x7f25b4105ae0 2024-09-21T01:29:46.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b419ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47270/0 (socket says 172.21.15.39:47270) 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 -- 172.21.15.39:0/1449299531 learned_addr learned my addr 172.21.15.39:0/1449299531 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b9400640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f25b4105ae0 0x7f25b419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b9e00640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b410bd60 0x7f25b41a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 -- 172.21.15.39:0/1449299531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f25b4105ae0 msgr2=0x7f25b419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f25b4105ae0 0x7f25b419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 -- 172.21.15.39:0/1449299531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b410bd60 msgr2=0x7f25b41a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b410bd60 0x7f25b41a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 -- 172.21.15.39:0/1449299531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b41a2420 con 0x7f25b410b410 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.394+0000 7f25b8a00640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b419ca90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f25a0005ab0 tx=0x7f25a0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.395+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a003f070 con 0x7f25b410b410 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.395+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25a0031b50 con 0x7f25b410b410 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.395+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0035030 con 0x7f25b410b410 2024-09-21T01:29:46.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.395+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b41a26b0 con 0x7f25b410b410 2024-09-21T01:29:46.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.395+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b41a2b50 con 0x7f25b410b410 2024-09-21T01:29:46.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.396+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b4105ef0 con 0x7f25b410b410 2024-09-21T01:29:46.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.398+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f25a0042070 con 0x7f25b410b410 2024-09-21T01:29:46.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.400+0000 7f25b2000640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f259c0b6700 0x7f259c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:46.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.400+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f25a01084b0 con 0x7f25b410b410 2024-09-21T01:29:46.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.400+0000 7f25b9400640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f259c0b6700 0x7f259c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:46.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.401+0000 7f25b9400640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f259c0b6700 0x7f259c0b8c10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009bd0 tx=0x7f25a8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:46.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.402+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f25b41a2b50 con 0x7f25b410b410 2024-09-21T01:29:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.655+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f25b419d610 con 0x7f25b410b410 2024-09-21T01:29:46.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.666+0000 7f25b2000640 1 -- 172.21.15.39:0/1449299531 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v24) ==== 81+0+834054 (secure 0 0 0) 0x7f25b419d610 con 0x7f25b410b410 2024-09-21T01:29:46.677 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 24, 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "active_gid": 14223, 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "active_name": "x", 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "active_addrs": { 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6800", 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2923702906 2024-09-21T01:29:46.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6801", 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2923702906 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "active_addr": "172.21.15.39:6801/2923702906", 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "active_change": "2024-09-21T00:53:07.774427+0000", 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "active_mgr_features": 4540719139924082687, 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "available": true, 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "standbys": [ 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.679 INFO:teuthology.orchestra.run.smithi039.stdout: "gid": 24169, 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "z", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_features": 4540719139924082687, 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "available_modules": [ 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "alerts", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:46.680 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.681 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.682 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.683 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.684 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_destination": { 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_destination", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.686 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_from_name": { 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_from_name", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.687 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "Ceph", 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email From: name", 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_host": { 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_host", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP server", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.689 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_password": { 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_password", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Password to authenticate with", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "465", 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.691 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP port", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_sender": { 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_sender", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.692 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP envelope sender", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_ssl": { 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_ssl", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.693 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_user": { 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_user", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "User to authenticate as", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.695 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "balancer", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.696 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "active": { 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "active", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.697 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_time": { 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_time", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0000", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_weekday": { 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_weekday", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_max_iterations": { 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_max_iterations", 2024-09-21T01:29:46.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "25", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "250", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_metrics": { 2024-09-21T01:29:46.700 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_metrics", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_step": { 2024-09-21T01:29:46.701 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_step", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.5", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "aggressiveness of optimization", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.702 INFO:teuthology.orchestra.run.smithi039.stdout: "end_time": { 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_time", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2359", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: "end_weekday": { 2024-09-21T01:29:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_weekday", 2024-09-21T01:29:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.706 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.707 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.708 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.709 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.710 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.711 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "upmap", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.712 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-compat", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "none", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "read", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap-read" 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Balancer mode", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.713 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_ids": { 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_ids", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:46.714 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_deviation": { 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_deviation", 2024-09-21T01:29:46.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_optimizations": { 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_optimizations", 2024-09-21T01:29:46.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.718 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_down_multiplier": { 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_down_multiplier", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T01:29:46.719 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_refresh_rate": { 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_refresh_rate", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "20", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.720 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_starting_port": { 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_starting_port", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4721", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.721 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_ptrace": { 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_ptrace", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.722 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_interval": { 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_interval", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:46.723 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_memory_target_ratio": { 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.724 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.7", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_log_destination": { 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm_log_destination", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.725 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "file", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "file,syslog", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "syslog" 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "cgroups_split": { 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cgroups_split", 2024-09-21T01:29:46.726 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "config_checks_enabled": { 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_checks_enabled", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "config_dashboard": { 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_dashboard", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.728 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_alertmanager": { 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_alertmanager", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.729 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_base": { 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_base", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.730 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Container image name, without the tag", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_elasticsearch": { 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_elasticsearch", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.731 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "elasticsearch container image", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_grafana": { 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_grafana", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.732 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_haproxy": { 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_haproxy", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.733 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "HAproxy container image", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_agent": { 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_agent", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.734 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger agent container image", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_collector": { 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_collector", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T01:29:46.735 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger collector container image", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_query": { 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_query", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.736 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger query container image", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_keepalived": { 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_keepalived", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T01:29:46.737 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Keepalived container image", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_loki": { 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_loki", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.738 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Loki container image", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nginx": { 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nginx", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.739 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nginx container image", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_node_exporter": { 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_node_exporter", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.740 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nvmeof": { 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nvmeof", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.741 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nvme-of container image", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_oauth2_proxy": { 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.742 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "oauth2-proxy container image", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_prometheus": { 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_prometheus", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.743 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_promtail": { 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_promtail", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.744 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Promtail container image", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_samba": { 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_samba", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.745 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Samba/SMB container image", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_samba_metrics": { 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_samba_metrics", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T01:29:46.746 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_snmp_gateway": { 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_snmp_gateway", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.747 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SNMP Gateway container image", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "container_init": { 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_init", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.748 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "daemon_cache_timeout": { 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "daemon_cache_timeout", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T01:29:46.749 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "default_cephadm_command_timeout": { 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "900", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.750 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_registry": { 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_registry", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.751 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "device_cache_timeout": { 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_cache_timeout", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1800", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache device inventory", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.752 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "device_enhanced_scan": { 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_enhanced_scan", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "facts_cache_timeout": { 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "facts_cache_timeout", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache host facts data", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "grafana_dashboards_path": { 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "grafana_dashboards_path", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "host_check_interval": { 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "host_check_interval", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to perform a host check", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.756 INFO:teuthology.orchestra.run.smithi039.stdout: "hw_monitoring": { 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hw_monitoring", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "inventory_list_all": { 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "inventory_list_all", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.758 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.759 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "log_refresh_metadata": { 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_refresh_metadata", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.760 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.761 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.762 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.763 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T01:29:46.764 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "*", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "max_count_per_host": { 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_count_per_host", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.766 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of daemons per service per host", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd_draining_count": { 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_osd_draining_count", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.767 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "migration_current": { 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "migration_current", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "internal - do not modify", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.769 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "root", 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm-package", 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "root" 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "oob_default_addr": { 2024-09-21T01:29:46.770 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "oob_default_addr", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "169.254.1.1", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.771 INFO:teuthology.orchestra.run.smithi039.stdout: "prometheus_alerts_path": { 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus_alerts_path", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.772 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_insecure": { 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_insecure", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_password": { 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_password", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T01:29:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_url": { 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_url", 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_username": { 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_username", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "secure_monitoring_stack": { 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secure_monitoring_stack", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.777 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "service_discovery_port": { 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "service_discovery_port", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8765", 2024-09-21T01:29:46.778 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "cephadm service discovery port", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.780 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_keepalive_count_max": { 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.781 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_keepalive_interval": { 2024-09-21T01:29:46.782 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_keepalive_interval", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "use_agent": { 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_agent", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "use_repo_digest": { 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_repo_digest", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.785 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_failed_host_check": { 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_failed_host_check", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.786 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_daemons": { 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_daemons", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_hosts": { 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_hosts", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crash", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.789 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.790 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.791 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.792 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.793 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "retain_interval": { 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retain_interval", 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.794 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "31536000", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.795 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_recent_interval": { 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_recent_interval", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.796 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1209600", 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dashboard", 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.797 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.798 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_ENABLED": { 2024-09-21T01:29:46.800 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T01:29:46.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T01:29:46.802 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T01:29:46.803 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T01:29:46.804 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.805 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.806 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T01:29:46.807 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.808 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.809 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.810 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.811 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_URL": { 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_URL", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.813 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.814 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.815 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.816 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.817 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "[]", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.819 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "{}", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.820 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.821 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.822 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.826 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.827 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.828 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.832 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "45", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.835 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.836 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.838 INFO:teuthology.orchestra.run.smithi039.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.839 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.840 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "cross_origin_url": { 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cross_origin_url", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.843 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "crt_file": { 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crt_file", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "debug": { 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "debug", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable/disable debug options", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.845 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "jwt_token_ttl": { 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "jwt_token_ttl", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "28800", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.846 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.847 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.848 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.849 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.850 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.851 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "motd": { 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "motd", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "The message of the day", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect_resolve_ip_addr": { 2024-09-21T01:29:46.853 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:46.854 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:46.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8080", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.856 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.857 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8443", 2024-09-21T01:29:46.858 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "sso_oauth2": { 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sso_oauth2", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.859 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "redirect", 2024-09-21T01:29:46.860 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect" 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.861 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "url_prefix": { 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url_prefix", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.863 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_monitoring": { 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_monitoring", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "monitor device health metrics", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.864 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.865 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.866 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.867 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.868 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "mark_out_threshold": { 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mark_out_threshold", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2419200", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": { 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_name", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "device_health_metrics", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "retention_period": { 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retention_period", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.871 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15552000", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain device health metrics", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_frequency": { 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_frequency", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.872 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "self_heal": { 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "self_heal", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.873 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.874 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.875 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_threshold": { 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_threshold", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7257600", 2024-09-21T01:29:46.876 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_local", 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.877 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.878 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.879 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.880 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.881 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "predict_interval": { 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predict_interval", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-09-21T01:29:46.882 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "predictor_model": { 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predictor_model", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "prophetstor", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.883 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.884 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.885 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "influx", 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": false, 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "influxdb python module not found", 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "batch_size": { 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "batch_size", 2024-09-21T01:29:46.886 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5000", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "database": { 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "database", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "ceph", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": { 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hostname", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server hostname", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.889 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "5", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.890 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.891 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.892 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.893 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.895 INFO:teuthology.orchestra.run.smithi039.stdout: "password": { 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "password", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "password of InfluxDB server user", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.896 INFO:teuthology.orchestra.run.smithi039.stdout: "port": { 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "port", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8086", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server port", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.897 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.898 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "false", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "threads": { 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threads", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "32", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "username of InfluxDB server user", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "verify_ssl": { 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "verify_ssl", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "true", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.902 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "insights", 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.905 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.906 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.907 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.908 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.909 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.909 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.911 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.912 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.913 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.914 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.915 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "k8sevents", 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_event_retention_days": { 2024-09-21T01:29:46.916 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ceph_event_retention_days", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "config_check_secs": { 2024-09-21T01:29:46.917 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_check_secs", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "10", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.918 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.919 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.920 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.921 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.922 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.923 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "localpool", 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.924 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "failure_domain": { 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "failure_domain", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "host", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "failure domain for any created local pool", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.925 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.926 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.927 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.928 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.930 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": { 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_size", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default min_size for any created local pool", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T01:29:46.931 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "num_rep": { 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "num_rep", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default replica count for any created local pool", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.932 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": { 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_num", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "128", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prefix", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name prefix for any created local pool", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.934 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "subtree": { 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "subtree", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "rack", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mds_autoscaler", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.937 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.938 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.939 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.940 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.941 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.942 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirroring", 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.943 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.944 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.946 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.947 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.948 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "nfs", 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.949 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.950 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.951 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.952 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.953 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.954 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.955 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "fail_fs": { 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "fail_fs", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.956 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.957 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.958 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.959 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.960 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.961 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "rook", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "test_orchestrator" 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Orchestrator backend", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.962 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.963 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_perf_query", 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.964 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.965 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.966 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.967 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.968 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.969 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_support", 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.970 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.971 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.972 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.973 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.975 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.976 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_autoscaler", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.977 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.979 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:46.981 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.982 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "threshold": { 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threshold", 2024-09-21T01:29:46.983 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1.0", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "scaling threshold", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "progress", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_pg_recovery_event": { 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_pg_recovery_event", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.985 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.986 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.987 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.988 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:46.989 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.990 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.991 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "max_completed_events": { 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_completed_events", 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.992 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "50", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "number of past completed events to remember", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.993 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long the module is going to sleep", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:46.994 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:46.995 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "cache": { 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cache", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.996 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "exclude_perf_counters": { 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "exclude_perf_counters", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T01:29:46.997 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:46.998 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:46.999 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.000 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.001 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.002 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools": { 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.003 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "300", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_interval": { 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_interval", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15.0", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T01:29:47.006 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "9283", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.007 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.008 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "stale_cache_strategy": { 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stale_cache_strategy", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "log", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.009 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "default", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "default", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "error" 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.010 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.011 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.012 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.014 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.015 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.016 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_snap_create": { 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_snap_create", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.017 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "mirror_snapshot_schedule": { 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.018 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.019 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "trash_purge_schedule": { 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "trash_purge_schedule", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.020 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "restful", 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_auth": { 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_auth", 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.021 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.022 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.023 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.026 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.027 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "max_requests": { 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_requests", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.028 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.030 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rgw", 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.032 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.033 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.034 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.035 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.036 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.037 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.038 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rook", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.039 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.040 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.042 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.043 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "storage_class": { 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "storage_class", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "local", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.045 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "selftest", 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.046 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.047 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.048 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.049 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.050 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.051 INFO:teuthology.orchestra.run.smithi039.stdout: "roption1": { 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption1", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.052 INFO:teuthology.orchestra.run.smithi039.stdout: "roption2": { 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption2", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "xyz", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption1": { 2024-09-21T01:29:47.053 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption1", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption2": { 2024-09-21T01:29:47.054 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption2", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption3": { 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption3", 2024-09-21T01:29:47.055 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption4": { 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption4", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.056 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption5": { 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption5", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.057 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption6": { 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption6", 2024-09-21T01:29:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption7": { 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption7", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "42", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.060 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "testkey": { 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testkey", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.061 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "testlkey": { 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testlkey", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.062 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testnewline", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.063 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smb", 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.064 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "internal_store_backend": { 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "internal_store_backend", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.065 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.066 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.067 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.068 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.069 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.070 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "update_orchestration": { 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "update_orchestration", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.071 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_m_granularity": { 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_m_granularity", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.073 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "dump_on_update": { 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dump_on_update", 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.074 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "dump database to debug log on update", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.075 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.076 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.077 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.078 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.079 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.080 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stats", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.081 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.082 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.083 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "status", 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.087 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.088 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.089 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.090 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.091 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.092 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telegraf", 2024-09-21T01:29:47.093 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "address": { 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "address", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.094 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15", 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.095 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.096 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.098 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.099 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.101 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telemetry", 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_basic": { 2024-09-21T01:29:47.102 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_basic", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_crash": { 2024-09-21T01:29:47.103 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_crash", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_device": { 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_device", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.104 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_ident": { 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_ident", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_perf": { 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_perf", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.106 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "contact": { 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "contact", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.107 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "description": { 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "description", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.108 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "device_url": { 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_url", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T01:29:47.109 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.110 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "24", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "8", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.111 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.112 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.113 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.113 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.113 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "last_opt_revision": { 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "last_opt_revision", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard_description": { 2024-09-21T01:29:47.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard_description", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.116 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.117 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.118 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.119 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.120 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "organization": { 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "organization", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.121 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "proxy": { 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "proxy", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.122 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.123 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "url": { 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.124 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "test_orchestrator", 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.125 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.126 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.127 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.128 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.129 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.132 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.134 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.135 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_clones": { 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_clones", 2024-09-21T01:29:47.136 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "periodic_async_work": { 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "periodic_async_work", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.137 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Periodically check for async work", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_delay": { 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_delay", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.138 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_no_wait": { 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.139 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.140 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "discovery_interval": { 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "discovery_interval", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "100", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.142 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "identifier": { 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "identifier", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.143 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.145 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.146 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.147 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.149 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_host": { 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_host", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.150 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_port": { 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_port", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10051", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_sender": { 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_sender", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.152 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "gid": 24173, 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "y", 2024-09-21T01:29:47.153 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_features": 4540719139924082687, 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "available_modules": [ 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "alerts", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.154 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.155 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.156 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.159 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_destination": { 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_destination", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.160 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_from_name": { 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_from_name", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.161 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "Ceph", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email From: name", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_host": { 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_host", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.162 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP server", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_password": { 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_password", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.163 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Password to authenticate with", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "465", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.164 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP port", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_sender": { 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_sender", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP envelope sender", 2024-09-21T01:29:47.165 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_ssl": { 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_ssl", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.166 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_user": { 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_user", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.167 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "User to authenticate as", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.168 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "balancer", 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "active": { 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "active", 2024-09-21T01:29:47.169 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_time": { 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_time", 2024-09-21T01:29:47.170 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0000", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_weekday": { 2024-09-21T01:29:47.171 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_weekday", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_max_iterations": { 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_max_iterations", 2024-09-21T01:29:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "25", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "250", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_metrics": { 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_metrics", 2024-09-21T01:29:47.173 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_step": { 2024-09-21T01:29:47.174 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_step", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.5", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "aggressiveness of optimization", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.175 INFO:teuthology.orchestra.run.smithi039.stdout: "end_time": { 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_time", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2359", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "end_weekday": { 2024-09-21T01:29:47.176 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_weekday", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.177 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.178 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.179 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.180 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.181 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { 2024-09-21T01:29:47.182 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-09-21T01:29:47.183 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "upmap", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-compat", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "none", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "read", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap-read" 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Balancer mode", 2024-09-21T01:29:47.184 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_ids": { 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_ids", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.187 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_deviation": { 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_deviation", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_optimizations": { 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_optimizations", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.189 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm", 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_down_multiplier": { 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_down_multiplier", 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-09-21T01:29:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_refresh_rate": { 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_refresh_rate", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.191 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "20", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_starting_port": { 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_starting_port", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4721", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_ptrace": { 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_ptrace", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.193 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_interval": { 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_interval", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.194 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_memory_target_ratio": { 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.195 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.7", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_log_destination": { 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm_log_destination", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.196 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "file", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "file,syslog", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "syslog" 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "cgroups_split": { 2024-09-21T01:29:47.197 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cgroups_split", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "config_checks_enabled": { 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_checks_enabled", 2024-09-21T01:29:47.198 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "config_dashboard": { 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_dashboard", 2024-09-21T01:29:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_alertmanager": { 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_alertmanager", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.200 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_base": { 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_base", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.201 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Container image name, without the tag", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_elasticsearch": { 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_elasticsearch", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.202 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "elasticsearch container image", 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_grafana": { 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_grafana", 2024-09-21T01:29:47.203 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_haproxy": { 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_haproxy", 2024-09-21T01:29:47.204 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "HAproxy container image", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_agent": { 2024-09-21T01:29:47.205 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_agent", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger agent container image", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.206 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_collector": { 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_collector", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger collector container image", 2024-09-21T01:29:47.207 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_query": { 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_query", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger query container image", 2024-09-21T01:29:47.208 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_keepalived": { 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_keepalived", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.209 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Keepalived container image", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_loki": { 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_loki", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.210 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Loki container image", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nginx": { 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nginx", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.211 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nginx container image", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_node_exporter": { 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_node_exporter", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.212 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nvmeof": { 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nvmeof", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nvme-of container image", 2024-09-21T01:29:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_oauth2_proxy": { 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.214 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "oauth2-proxy container image", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_prometheus": { 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_prometheus", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.215 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_promtail": { 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_promtail", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.216 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Promtail container image", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_samba": { 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_samba", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.217 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Samba/SMB container image", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_samba_metrics": { 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_samba_metrics", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.218 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_snmp_gateway": { 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_snmp_gateway", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.219 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SNMP Gateway container image", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "container_init": { 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_init", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.220 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "daemon_cache_timeout": { 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "daemon_cache_timeout", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T01:29:47.221 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "default_cephadm_command_timeout": { 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "900", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T01:29:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "default_registry": { 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_registry", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "device_cache_timeout": { 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_cache_timeout", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1800", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.224 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache device inventory", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "device_enhanced_scan": { 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_enhanced_scan", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.225 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "facts_cache_timeout": { 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "facts_cache_timeout", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.226 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache host facts data", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "grafana_dashboards_path": { 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "grafana_dashboards_path", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T01:29:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "host_check_interval": { 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "host_check_interval", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to perform a host check", 2024-09-21T01:29:47.228 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "hw_monitoring": { 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hw_monitoring", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.229 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "inventory_list_all": { 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "inventory_list_all", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.230 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.231 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "log_refresh_metadata": { 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_refresh_metadata", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.233 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.234 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.235 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.236 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "*", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.238 INFO:teuthology.orchestra.run.smithi039.stdout: "max_count_per_host": { 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_count_per_host", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of daemons per service per host", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd_draining_count": { 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_osd_draining_count", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "migration_current": { 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "migration_current", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "internal - do not modify", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-09-21T01:29:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "root", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm-package", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "root" 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "oob_default_addr": { 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "oob_default_addr", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "169.254.1.1", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "prometheus_alerts_path": { 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus_alerts_path", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.244 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_insecure": { 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_insecure", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_password": { 2024-09-21T01:29:47.245 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_password", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_url": { 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_url", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_username": { 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_username", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "secure_monitoring_stack": { 2024-09-21T01:29:47.248 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secure_monitoring_stack", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "service_discovery_port": { 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "service_discovery_port", 2024-09-21T01:29:47.249 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8765", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "cephadm service discovery port", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_keepalive_count_max": { 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_keepalive_interval": { 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_keepalive_interval", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "use_agent": { 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_agent", 2024-09-21T01:29:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "use_repo_digest": { 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_repo_digest", 2024-09-21T01:29:47.255 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_failed_host_check": { 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_failed_host_check", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.256 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_daemons": { 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_daemons", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.257 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_hosts": { 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_hosts", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.258 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crash", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.259 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.260 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.261 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.262 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.263 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "retain_interval": { 2024-09-21T01:29:47.264 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retain_interval", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "31536000", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.265 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_recent_interval": { 2024-09-21T01:29:47.266 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_recent_interval", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1209600", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dashboard", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.269 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.270 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_ENABLED": { 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.272 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.273 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.274 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.275 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.276 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.277 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.278 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.279 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.280 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.281 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.283 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_URL": { 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_URL", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.287 INFO:teuthology.orchestra.run.smithi039.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T01:29:47.288 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T01:29:47.289 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "[]", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T01:29:47.290 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "{}", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.291 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T01:29:47.292 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T01:29:47.293 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T01:29:47.294 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T01:29:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.297 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.298 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.299 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.300 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.301 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.302 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8", 2024-09-21T01:29:47.303 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "45", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.304 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.305 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.306 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.307 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.308 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.310 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.311 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.312 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "cross_origin_url": { 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cross_origin_url", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.313 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "crt_file": { 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crt_file", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.314 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "debug": { 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "debug", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable/disable debug options", 2024-09-21T01:29:47.315 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.316 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.316 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.316 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.316 INFO:teuthology.orchestra.run.smithi039.stdout: "jwt_token_ttl": { 2024-09-21T01:29:47.316 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "jwt_token_ttl", 2024-09-21T01:29:47.317 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.317 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "28800", 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.318 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.319 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.320 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.321 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.322 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.325 INFO:teuthology.orchestra.run.smithi039.stdout: "motd": { 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "motd", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "The message of the day", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect_resolve_ip_addr": { 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.327 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.328 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8080", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.329 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.330 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.331 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8443", 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "sso_oauth2": { 2024-09-21T01:29:47.332 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sso_oauth2", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-09-21T01:29:47.333 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "redirect", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect" 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.334 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.335 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "url_prefix": { 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url_prefix", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.336 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_monitoring": { 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_monitoring", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "monitor device health metrics", 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.338 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.339 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.340 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.341 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.342 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "mark_out_threshold": { 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mark_out_threshold", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2419200", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.343 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": { 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_name", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "device_health_metrics", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T01:29:47.344 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "retention_period": { 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retention_period", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15552000", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain device health metrics", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.345 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_frequency": { 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_frequency", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.346 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "self_heal": { 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "self_heal", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.347 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.349 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_threshold": { 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_threshold", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7257600", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_local", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.352 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.353 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.354 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "predict_interval": { 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predict_interval", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.356 INFO:teuthology.orchestra.run.smithi039.stdout: "predictor_model": { 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predictor_model", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "prophetstor", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.357 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.358 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.359 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "influx", 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": false, 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "influxdb python module not found", 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "batch_size": { 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "batch_size", 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.360 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5000", 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.361 INFO:teuthology.orchestra.run.smithi039.stdout: "database": { 2024-09-21T01:29:47.362 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "database", 2024-09-21T01:29:47.362 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.362 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.362 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.362 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "ceph", 2024-09-21T01:29:47.362 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": { 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hostname", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.363 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server hostname", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "5", 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.366 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.367 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.368 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.369 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "password": { 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "password", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.370 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "password of InfluxDB server user", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "port": { 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "port", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.371 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.372 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8086", 2024-09-21T01:29:47.372 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.372 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.372 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.372 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server port", 2024-09-21T01:29:47.372 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.373 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.374 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "false", 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "threads": { 2024-09-21T01:29:47.375 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threads", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "32", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "username of InfluxDB server user", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.377 INFO:teuthology.orchestra.run.smithi039.stdout: "verify_ssl": { 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "verify_ssl", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "true", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "insights", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.379 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.380 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.381 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.382 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.383 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.384 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.385 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.386 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.387 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.388 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.389 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.390 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "k8sevents", 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.391 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_event_retention_days": { 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ceph_event_retention_days", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "config_check_secs": { 2024-09-21T01:29:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_check_secs", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "10", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.393 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.394 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.395 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.397 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.398 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "localpool", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "failure_domain": { 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "failure_domain", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "host", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "failure domain for any created local pool", 2024-09-21T01:29:47.400 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.401 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.402 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.403 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.405 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.409 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.409 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.409 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.409 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": { 2024-09-21T01:29:47.409 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_size", 2024-09-21T01:29:47.410 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.410 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.410 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.410 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.410 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.410 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.411 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.411 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default min_size for any created local pool", 2024-09-21T01:29:47.411 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T01:29:47.411 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.411 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "num_rep": { 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "num_rep", 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-09-21T01:29:47.412 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default replica count for any created local pool", 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": { 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_num", 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "128", 2024-09-21T01:29:47.414 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.415 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.415 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.415 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T01:29:47.415 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.415 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.415 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.416 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { 2024-09-21T01:29:47.416 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prefix", 2024-09-21T01:29:47.416 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.416 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.416 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name prefix for any created local pool", 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.417 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.418 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.418 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.418 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.418 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.418 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.419 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "subtree": { 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "subtree", 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.421 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.421 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "rack", 2024-09-21T01:29:47.421 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.421 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.421 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.421 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T01:29:47.422 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T01:29:47.422 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.422 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.422 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.422 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mds_autoscaler", 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.423 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.424 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.425 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.425 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.425 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.425 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.425 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.425 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.426 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.427 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.428 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.429 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.430 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.431 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.432 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirroring", 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.435 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.436 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.437 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.438 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "nfs", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.440 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.441 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.442 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.443 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.445 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "fail_fs": { 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "fail_fs", 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.446 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.448 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.449 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.450 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.451 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.452 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "rook", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "test_orchestrator" 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Orchestrator backend", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.453 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.454 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_perf_query", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.455 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.456 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.458 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.459 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.460 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.462 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_support", 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.463 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.464 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.465 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.466 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.467 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.469 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_autoscaler", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.471 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.472 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.473 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.474 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.476 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.477 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "threshold": { 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threshold", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1.0", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "scaling threshold", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T01:29:47.478 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "progress", 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_pg_recovery_event": { 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_pg_recovery_event", 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.479 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.480 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.481 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.482 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.483 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.484 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "max_completed_events": { 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_completed_events", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "50", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "number of past completed events to remember", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.487 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long the module is going to sleep", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.488 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus", 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "cache": { 2024-09-21T01:29:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cache", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "exclude_perf_counters": { 2024-09-21T01:29:47.490 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "exclude_perf_counters", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.491 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.492 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.493 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.494 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools": { 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.496 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "300", 2024-09-21T01:29:47.497 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_interval": { 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_interval", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15.0", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.499 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "9283", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.500 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.501 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "stale_cache_strategy": { 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stale_cache_strategy", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "log", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "default", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "default", 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "error" 2024-09-21T01:29:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.504 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.505 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.506 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.507 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.508 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.509 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_snap_create": { 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_snap_create", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.510 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "mirror_snapshot_schedule": { 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.511 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.512 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "trash_purge_schedule": { 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "trash_purge_schedule", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.513 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "restful", 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.514 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_auth": { 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_auth", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.515 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.517 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.518 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.519 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.520 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.520 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "max_requests": { 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_requests", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.523 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.524 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.525 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rgw", 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.526 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.527 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.528 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.529 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T01:29:47.531 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rook", 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.533 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.534 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.535 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.536 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.538 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "storage_class": { 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "storage_class", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.539 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "local", 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.540 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "selftest", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.541 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.542 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.543 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.545 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "roption1": { 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption1", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.546 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "roption2": { 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption2", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "xyz", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.547 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption1": { 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption1", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.548 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption2": { 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption2", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.549 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption3": { 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption3", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.550 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption4": { 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption4", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.551 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption5": { 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption5", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption6": { 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption6", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption7": { 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption7", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "42", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.555 INFO:teuthology.orchestra.run.smithi039.stdout: "testkey": { 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testkey", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "testlkey": { 2024-09-21T01:29:47.556 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testlkey", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testnewline", 2024-09-21T01:29:47.557 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smb", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "internal_store_backend": { 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "internal_store_backend", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T01:29:47.559 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.560 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.561 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.562 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.564 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "update_orchestration": { 2024-09-21T01:29:47.565 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "update_orchestration", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_m_granularity": { 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_m_granularity", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.567 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "dump_on_update": { 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dump_on_update", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "dump database to debug log on update", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.569 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.570 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.574 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stats", 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.575 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.576 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.577 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.578 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.579 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.580 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "status", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.582 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.583 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.584 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.586 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.587 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telegraf", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "address": { 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "address", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.588 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.589 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.590 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.591 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.592 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.593 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.594 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.595 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telemetry", 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_basic": { 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_basic", 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.596 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_crash": { 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_crash", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.597 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_device": { 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_device", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.598 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_ident": { 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_ident", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_perf": { 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_perf", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.600 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "contact": { 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "contact", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.601 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "description": { 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "description", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.602 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "device_url": { 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_url", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.603 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.604 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "24", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "8", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.605 INFO:teuthology.orchestra.run.smithi039.stdout: "last_opt_revision": { 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "last_opt_revision", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard_description": { 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard_description", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.608 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.609 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.611 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "organization": { 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "organization", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "proxy": { 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "proxy", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.614 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.615 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "url": { 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.616 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "test_orchestrator", 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.617 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.618 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.619 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.620 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.621 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.622 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.623 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.624 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.625 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.627 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.628 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_clones": { 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_clones", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "periodic_async_work": { 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "periodic_async_work", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Periodically check for async work", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_delay": { 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_delay", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.631 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_no_wait": { 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.632 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix", 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "discovery_interval": { 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "discovery_interval", 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "100", 2024-09-21T01:29:47.634 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "identifier": { 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "identifier", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.635 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.636 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.637 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.638 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.639 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.641 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_host": { 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_host", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.643 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_port": { 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_port", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10051", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_sender": { 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_sender", 2024-09-21T01:29:47.644 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.645 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "modules": [ 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "dashboard", 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "iostat", 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "nfs", 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "restful", 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_schedule" 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "available_modules": [ 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "alerts", 2024-09-21T01:29:47.646 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.647 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.648 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.649 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.650 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.651 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_destination": { 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_destination", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.652 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_from_name": { 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_from_name", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.653 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "Ceph", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email From: name", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_host": { 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_host", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP server", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_password": { 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_password", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.655 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Password to authenticate with", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "465", 2024-09-21T01:29:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP port", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_sender": { 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_sender", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.657 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP envelope sender", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_ssl": { 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_ssl", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.658 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_user": { 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_user", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "User to authenticate as", 2024-09-21T01:29:47.659 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.660 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "balancer", 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "active": { 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "active", 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.661 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_time": { 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_time", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.662 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0000", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_weekday": { 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_weekday", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.663 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_max_iterations": { 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_max_iterations", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.664 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "25", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "250", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_metrics": { 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_metrics", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_step": { 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_step", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.666 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.5", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "aggressiveness of optimization", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "end_time": { 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_time", 2024-09-21T01:29:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2359", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "end_weekday": { 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_weekday", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.669 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.670 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.672 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.673 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T01:29:47.674 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "upmap", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-compat", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "none", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "read", 2024-09-21T01:29:47.675 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap-read" 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Balancer mode", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_ids": { 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_ids", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.676 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.677 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.678 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_deviation": { 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_deviation", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T01:29:47.679 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_optimizations": { 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_optimizations", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T01:29:47.680 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm", 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_down_multiplier": { 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_down_multiplier", 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_refresh_rate": { 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_refresh_rate", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "20", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_starting_port": { 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_starting_port", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.683 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4721", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_ptrace": { 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_ptrace", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.684 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_interval": { 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_interval", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.685 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_memory_target_ratio": { 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.7", 2024-09-21T01:29:47.686 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_log_destination": { 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm_log_destination", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.687 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "file", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "file,syslog", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "syslog" 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cgroups_split": { 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cgroups_split", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.688 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "config_checks_enabled": { 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_checks_enabled", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.689 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "config_dashboard": { 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_dashboard", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.690 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_alertmanager": { 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_alertmanager", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T01:29:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_base": { 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_base", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.692 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Container image name, without the tag", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_elasticsearch": { 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_elasticsearch", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.693 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "elasticsearch container image", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_grafana": { 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_grafana", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.694 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_haproxy": { 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_haproxy", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "HAproxy container image", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_agent": { 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_agent", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.696 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger agent container image", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_collector": { 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_collector", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.697 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger collector container image", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_query": { 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_query", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger query container image", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_keepalived": { 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_keepalived", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Keepalived container image", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_loki": { 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_loki", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Loki container image", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.700 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nginx": { 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nginx", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nginx container image", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_node_exporter": { 2024-09-21T01:29:47.701 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_node_exporter", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nvmeof": { 2024-09-21T01:29:47.702 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nvmeof", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nvme-of container image", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_oauth2_proxy": { 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T01:29:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "oauth2-proxy container image", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_prometheus": { 2024-09-21T01:29:47.704 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_prometheus", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_promtail": { 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_promtail", 2024-09-21T01:29:47.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Promtail container image", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_samba": { 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_samba", 2024-09-21T01:29:47.706 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Samba/SMB container image", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_samba_metrics": { 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_samba_metrics", 2024-09-21T01:29:47.707 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_snmp_gateway": { 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_snmp_gateway", 2024-09-21T01:29:47.708 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SNMP Gateway container image", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "container_init": { 2024-09-21T01:29:47.709 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_init", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "daemon_cache_timeout": { 2024-09-21T01:29:47.710 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "daemon_cache_timeout", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "default_cephadm_command_timeout": { 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T01:29:47.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "900", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "default_registry": { 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_registry", 2024-09-21T01:29:47.712 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "device_cache_timeout": { 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_cache_timeout", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1800", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache device inventory", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "device_enhanced_scan": { 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_enhanced_scan", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.714 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "facts_cache_timeout": { 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "facts_cache_timeout", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.715 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache host facts data", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "grafana_dashboards_path": { 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "grafana_dashboards_path", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.716 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "host_check_interval": { 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "host_check_interval", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.717 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to perform a host check", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "hw_monitoring": { 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hw_monitoring", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.718 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "inventory_list_all": { 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "inventory_list_all", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.719 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "log_refresh_metadata": { 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_refresh_metadata", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.725 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.726 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "*", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.728 INFO:teuthology.orchestra.run.smithi039.stdout: "max_count_per_host": { 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_count_per_host", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of daemons per service per host", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd_draining_count": { 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_osd_draining_count", 2024-09-21T01:29:47.729 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "migration_current": { 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "migration_current", 2024-09-21T01:29:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "internal - do not modify", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.731 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "root", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm-package", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "root" 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.732 INFO:teuthology.orchestra.run.smithi039.stdout: "oob_default_addr": { 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "oob_default_addr", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "169.254.1.1", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "prometheus_alerts_path": { 2024-09-21T01:29:47.733 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus_alerts_path", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_insecure": { 2024-09-21T01:29:47.734 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_insecure", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_password": { 2024-09-21T01:29:47.735 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_password", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_url": { 2024-09-21T01:29:47.736 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_url", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_username": { 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_username", 2024-09-21T01:29:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "secure_monitoring_stack": { 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secure_monitoring_stack", 2024-09-21T01:29:47.738 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "service_discovery_port": { 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "service_discovery_port", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.739 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8765", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "cephadm service discovery port", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.740 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.741 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_keepalive_count_max": { 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-09-21T01:29:47.742 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_keepalive_interval": { 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_keepalive_interval", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.743 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "use_agent": { 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_agent", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.744 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "use_repo_digest": { 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_repo_digest", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T01:29:47.745 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_failed_host_check": { 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_failed_host_check", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.746 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_daemons": { 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_daemons", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.747 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_hosts": { 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_hosts", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.748 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crash", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.749 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.750 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.751 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.752 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.753 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "retain_interval": { 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retain_interval", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.754 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "31536000", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.755 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_recent_interval": { 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_recent_interval", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1209600", 2024-09-21T01:29:47.756 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dashboard", 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T01:29:47.757 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T01:29:47.758 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T01:29:47.759 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_ENABLED": { 2024-09-21T01:29:47.760 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T01:29:47.761 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T01:29:47.762 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T01:29:47.763 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T01:29:47.768 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T01:29:47.769 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T01:29:47.770 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T01:29:47.771 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.772 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_URL": { 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_URL", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.773 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.775 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.776 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.777 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.778 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.779 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "[]", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.780 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "{}", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T01:29:47.781 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T01:29:47.782 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T01:29:47.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T01:29:47.784 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T01:29:47.785 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T01:29:47.786 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.787 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.788 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.789 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.790 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.791 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.792 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.793 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.794 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "45", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.795 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.796 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.797 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.798 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.799 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.800 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.801 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.802 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "cross_origin_url": { 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cross_origin_url", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "crt_file": { 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crt_file", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.805 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "debug": { 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "debug", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.806 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable/disable debug options", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "jwt_token_ttl": { 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "jwt_token_ttl", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "28800", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.807 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.808 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.809 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.813 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "motd": { 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "motd", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.814 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "The message of the day", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect_resolve_ip_addr": { 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.815 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.816 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8080", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.819 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { 2024-09-21T01:29:47.820 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8443", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "sso_oauth2": { 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sso_oauth2", 2024-09-21T01:29:47.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-09-21T01:29:47.822 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "redirect", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect" 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.823 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.824 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "url_prefix": { 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url_prefix", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.825 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_monitoring": { 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_monitoring", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.826 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "monitor device health metrics", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.827 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.828 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.830 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "mark_out_threshold": { 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mark_out_threshold", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.832 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2419200", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": { 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_name", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.833 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "device_health_metrics", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "retention_period": { 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retention_period", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.834 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15552000", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain device health metrics", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_frequency": { 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_frequency", 2024-09-21T01:29:47.835 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "self_heal": { 2024-09-21T01:29:47.836 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "self_heal", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.837 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.839 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_threshold": { 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_threshold", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7257600", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T01:29:47.840 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_local", 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.841 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.842 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.843 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.844 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "predict_interval": { 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predict_interval", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.846 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "predictor_model": { 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predictor_model", 2024-09-21T01:29:47.847 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "prophetstor", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.848 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.849 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.850 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "influx", 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": false, 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "influxdb python module not found", 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "batch_size": { 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "batch_size", 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5000", 2024-09-21T01:29:47.851 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "database": { 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "database", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.852 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "ceph", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": { 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hostname", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.853 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server hostname", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.854 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "5", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.855 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.856 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.857 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.858 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.859 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "password": { 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "password", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "password of InfluxDB server user", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.860 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "port": { 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "port", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8086", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server port", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.862 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "false", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "threads": { 2024-09-21T01:29:47.863 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threads", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "32", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", 2024-09-21T01:29:47.864 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "username of InfluxDB server user", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "verify_ssl": { 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "verify_ssl", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.865 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "true", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "insights", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.866 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.867 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.868 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.871 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.872 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.873 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.874 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.876 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.877 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.878 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "k8sevents", 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_event_retention_days": { 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ceph_event_retention_days", 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.879 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "config_check_secs": { 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_check_secs", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.880 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "10", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.881 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.882 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.884 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.885 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.886 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "localpool", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "failure_domain": { 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "failure_domain", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "host", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.888 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "failure domain for any created local pool", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.889 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.890 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.891 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.892 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.893 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": { 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_size", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.894 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default min_size for any created local pool", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "num_rep": { 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "num_rep", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.895 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default replica count for any created local pool", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": { 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_num", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.896 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "128", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prefix", 2024-09-21T01:29:47.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name prefix for any created local pool", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.898 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "subtree": { 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "subtree", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.899 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "rack", 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.900 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mds_autoscaler", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.901 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.902 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.903 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.904 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.905 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.906 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirroring", 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.907 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.908 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.909 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.910 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.911 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.912 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "nfs", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.915 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.916 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.917 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.918 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "fail_fs": { 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "fail_fs", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.920 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.921 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.922 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.923 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.924 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.925 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.926 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "rook", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "test_orchestrator" 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Orchestrator backend", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.927 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.928 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.928 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.929 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.929 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.929 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.929 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.929 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.929 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_perf_query", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.930 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.931 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.932 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.933 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.934 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.935 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_support", 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.936 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.937 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.938 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.939 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.940 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.941 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_autoscaler", 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.942 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.943 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.944 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.945 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.946 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.947 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.948 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "threshold": { 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threshold", 2024-09-21T01:29:47.949 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1.0", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "scaling threshold", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "progress", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_pg_recovery_event": { 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_pg_recovery_event", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.951 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.952 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.953 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.954 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.955 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.956 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.957 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "max_completed_events": { 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_completed_events", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "50", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "number of past completed events to remember", 2024-09-21T01:29:47.958 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long the module is going to sleep", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.959 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.960 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus", 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "cache": { 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cache", 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.961 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "exclude_perf_counters": { 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "exclude_perf_counters", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.962 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.963 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.964 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.965 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.966 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.967 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools": { 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.969 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "300", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_interval": { 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_interval", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.970 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15.0", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.971 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.972 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "9283", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.973 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "stale_cache_strategy": { 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stale_cache_strategy", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.974 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "log", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.975 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "default", 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "default", 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "error" 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-09-21T01:29:47.976 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.978 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.979 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.980 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.981 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_snap_create": { 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_snap_create", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.983 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "mirror_snapshot_schedule": { 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.984 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.985 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "trash_purge_schedule": { 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "trash_purge_schedule", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.986 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "restful", 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_auth": { 2024-09-21T01:29:47.987 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_auth", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-09-21T01:29:47.988 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.989 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.990 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.991 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:47.992 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:47.993 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "max_requests": { 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_requests", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.994 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-09-21T01:29:47.995 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-09-21T01:29:47.996 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:47.997 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:47.998 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rgw", 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:47.999 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.001 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.002 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.003 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-09-21T01:29:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.005 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.006 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rook", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.007 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.008 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.009 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.010 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.011 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: "storage_class": { 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "storage_class", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "local", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "selftest", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.014 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.015 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.016 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.017 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.018 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "roption1": { 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption1", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.019 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "roption2": { 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption2", 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "xyz", 2024-09-21T01:29:48.020 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption1": { 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption1", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.021 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption2": { 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption2", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.022 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption3": { 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption3", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.023 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption4": { 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption4", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.024 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption5": { 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption5", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.025 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption6": { 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption6", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.026 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption7": { 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption7", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "42", 2024-09-21T01:29:48.027 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.028 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "testkey": { 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testkey", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.029 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "testlkey": { 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testlkey", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.030 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testnewline", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.031 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smb", 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.032 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "internal_store_backend": { 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "internal_store_backend", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.034 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.035 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.036 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.037 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.038 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.039 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "update_orchestration": { 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "update_orchestration", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T01:29:48.040 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_m_granularity": { 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_m_granularity", 2024-09-21T01:29:48.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "dump_on_update": { 2024-09-21T01:29:48.042 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dump_on_update", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "dump database to debug log on update", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.044 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.045 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.046 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.047 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.048 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.049 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stats", 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.050 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.051 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.052 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.053 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.054 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.055 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.056 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "status", 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.058 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.059 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telegraf", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "address": { 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "address", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.064 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.066 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.067 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.069 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.070 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.071 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telemetry", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.072 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_basic": { 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_basic", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.073 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_crash": { 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_crash", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.074 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_device": { 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_device", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_ident": { 2024-09-21T01:29:48.075 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_ident", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_perf": { 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_perf", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.076 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "contact": { 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "contact", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.077 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "description": { 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "description", 2024-09-21T01:29:48.078 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "device_url": { 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_url", 2024-09-21T01:29:48.079 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-09-21T01:29:48.080 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.081 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "24", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "8", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "last_opt_revision": { 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "last_opt_revision", 2024-09-21T01:29:48.082 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.083 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard_description": { 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard_description", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.084 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.085 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.086 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.087 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.088 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.089 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "organization": { 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "organization", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.090 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "proxy": { 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "proxy", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.091 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.092 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "url": { 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.093 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "test_orchestrator", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.095 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.096 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.098 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.099 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.100 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.101 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.102 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.105 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.106 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_clones": { 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_clones", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.107 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "periodic_async_work": { 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "periodic_async_work", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.108 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Periodically check for async work", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_delay": { 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_delay", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.109 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_no_wait": { 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.110 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.111 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix", 2024-09-21T01:29:48.112 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "discovery_interval": { 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "discovery_interval", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "100", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.113 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "identifier": { 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "identifier", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.114 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-09-21T01:29:48.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.116 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.117 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-09-21T01:29:48.118 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.119 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "sqlite3_killpoint": { 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sqlite3_killpoint", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "dev", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-09-21T01:29:48.120 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_host": { 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_host", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.121 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_port": { 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_port", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10051", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.122 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_sender": { 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_sender", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-09-21T01:29:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "services": { 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "dashboard": "https://172.21.15.39:8443/" 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "always_on_modules": { 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus": [ 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-09-21T01:29:48.124 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific": [ 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-09-21T01:29:48.125 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "quincy": [ 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-09-21T01:29:48.126 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "reef": [ 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-09-21T01:29:48.127 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "squid": [ 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-09-21T01:29:48.128 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "last_failure_osd_epoch": 5, 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "active_clients": [ 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1426954663 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "libcephsqlite", 2024-09-21T01:29:48.129 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3729574193 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2883154799 2024-09-21T01:29:48.130 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1375529313 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.131 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-09-21T01:29:48.132 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:48.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.132 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:48.132 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3125538457 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T01:29:48.133 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2085490382 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T01:29:48.134 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:29:48.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f259c0b6700 msgr2=0x7f259c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f259c0b6700 0x7f259c0b8c10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009bd0 tx=0x7f25a8005f90 comp rx=0 tx=0).stop 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 msgr2=0x7f25b419ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b419ca90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f25a0005ab0 tx=0x7f25a0003b90 comp rx=0 tx=0).stop 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 shutdown_connections 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f259c0b6700 0x7f259c0b8c10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b410bd60 0x7f25b41a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25b410b410 0x7f25b419ca90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 --2- 172.21.15.39:0/1449299531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f25b4105ae0 0x7f25b419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.674+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 >> 172.21.15.39:0/1449299531 conn(0x7f25b4101560 msgr2=0x7f25b4108a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.675+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 shutdown_connections 2024-09-21T01:29:48.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:29:46.675+0000 7f25ba800640 1 -- 172.21.15.39:0/1449299531 wait complete. 2024-09-21T01:29:48.187 DEBUG:tasks.check_counter:Getting stats from a 2024-09-21T01:29:48.187 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.a perf dump 2024-09-21T01:29:48.187 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:46 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/40306426' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T01:29:48.187 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:46 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/1449299531' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T01:29:48.188 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:47 smithi039 ceph-mon[30415]: pgmap v1217: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 149 op/s 2024-09-21T01:29:48.188 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:46 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/40306426' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T01:29:48.189 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:46 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/1449299531' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T01:29:48.189 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:46 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/40306426' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T01:29:48.189 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:46 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/1449299531' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T01:29:48.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:47 smithi150 ceph-mon[34945]: pgmap v1217: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 149 op/s 2024-09-21T01:29:48.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:47 smithi160 ceph-mon[34696]: pgmap v1217: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 149 op/s 2024-09-21T01:29:48.578 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:49.013 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:48 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:29:49.307 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:29:49.307 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:29:49.307 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 273573, 2024-09-21T01:29:49.307 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 404066, 2024-09-21T01:29:49.307 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 86605923, 2024-09-21T01:29:49.307 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 534801047, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 15, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 7, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 76.090557094, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 61.662303773, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 29.429511724, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 2.080113852, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 404060, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 81.236889564, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000201051 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:29:49.308 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 320606, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.174078340, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000542 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 6720, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 32272 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 183182, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 167489, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 73391938, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 282408777, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 9, 2024-09-21T01:29:49.309 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 6, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 23.692282273, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 11.289586201, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 11.887401655, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 2.615560204, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 167491, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 26.564153458, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000158600 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 293000, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.103482838, 2024-09-21T01:29:49.310 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000353 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 1167312, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 303856 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 134326, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 131781, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 58137249, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 282054021, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 10, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 5, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 19.755147736, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 9.938102342, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 8.516105408, 2024-09-21T01:29:49.311 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 2.419362138, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 131781, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 28.856618018, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000218974 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 192948, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.065941764, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000341 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 711152 2024-09-21T01:29:49.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 1, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-MDSRank": { 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 59508, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 120.334295677, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.002022153 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-PurgeQueue": { 2024-09-21T01:29:49.313 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 38797, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 11.605226946, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000299126 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "request": 172312, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "reply": 165899, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "reply_latency": { 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 165899, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1331.242046409, 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.008024412 2024-09-21T01:29:49.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_reply": 0, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "forward": 6413, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_complete": 41, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_keys": 0, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_commit": 136, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_split": 11, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_merge": 1, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes": 2812, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_top": 807, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_bottom": 1727, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pin_tail": 278, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pinned": 1658, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_expired": 1302, 2024-09-21T01:29:49.315 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_with_caps": 237, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": 238, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "subtrees": 4, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse": 239903, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_hit": 150191, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_forward": 6395, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_discover": 55, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_dir_fetch": 0, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_remote_ino": 0, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_lock": 1565, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "load_cent": 24728, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "q": 0, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "exported": 49, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "exported_inodes": 1069, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "imported": 15, 2024-09-21T01:29:49.316 INFO:teuthology.orchestra.run.smithi039.stdout: "imported_inodes": 1076, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_dir_fetch": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_backtrace_fetch": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_peer_discover": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rfiles": 702, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rbytes": 212015681, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rsnaps": 6, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_set_tag": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_inotable_repaired": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_inodes": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_file_inodes": 0, 2024-09-21T01:29:49.317 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_inode_file_caps": 10, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_revoke": 17468, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_grant": 111089, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flushsnap_ack": 4, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps": 62184, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps_dirty": 20668, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_cap_release": 5110, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "process_request_cap_release": 122925 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache": { 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update": 32, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update_receipt": 45, 2024-09-21T01:29:49.318 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_try_discover": 30, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_send_discover": 85, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_handle_discover": 109, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays": 266, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_delayed": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_enqueuing": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_created": 23185, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_enqueued": 22919, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_reintegrated": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_migrated": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_processing": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_enqueued": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_prioritized": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_started": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_completed": 0, 2024-09-21T01:29:49.319 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_quiesce_path": 25, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_quiesce_inode": 3786, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_exportdir": 53, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_flush": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragmentdir": 12, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragstats": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_inodestats": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "uninline_started": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "uninline_succeeded": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "uninline_write_failed": 0 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log": { 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "evlrg": 0, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "evadd": 129789, 2024-09-21T01:29:49.320 INFO:teuthology.orchestra.run.smithi039.stdout: "evex": 54748, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "evtrm": 49161, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "ev": 80628, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "evexg": 0, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "evexd": 5587, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "segadd": 200, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "segex": 72, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "segtrm": 65, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "seg": 135, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "segmjr": 7, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "segexg": 0, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "segexd": 7, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "expos": 143699901, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "wrpos": 672707140, 2024-09-21T01:29:49.321 INFO:teuthology.orchestra.run.smithi039.stdout: "rdpos": 0, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "jlat": { 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 33977, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 354.524320279, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.010434244 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "replayed": 0 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mem": { 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "ino": 2363, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "ino+": 26433, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "ino-": 24070, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "dir": 143, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "dir+": 264, 2024-09-21T01:29:49.322 INFO:teuthology.orchestra.run.smithi039.stdout: "dir-": 121, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "dn": 2892, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "dn+": 50953, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "dn-": 48061, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "cap": 238, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "cap+": 26607, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "cap-": 26369, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "rss": 155188, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "heap": 368979968 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_server": { 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_client_request": 249966, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_server_request": 452, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_request": 172312, 2024-09-21T01:29:49.323 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_session": 3095, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_peer_request": 6149, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "req_create_latency": { 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 22883, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 363.168992225, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.015870689 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getattr_latency": { 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 230, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 2.774192802, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.012061707 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getfilelock_latency": { 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "req_link_latency": { 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookup_latency": { 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 37489, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 119.661311313, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.003191904 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookuphash_latency": { 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupino_latency": { 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000110128, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000110128 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupname_latency": { 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupparent_latency": { 2024-09-21T01:29:49.326 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.327 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:48 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:29:49.327 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.327 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.327 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.327 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupsnap_latency": { 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lssnap_latency": { 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mkdir_latency": { 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.328 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mknod_latency": { 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mksnap_latency": { 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "req_open_latency": { 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 2, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000737557, 2024-09-21T01:29:49.329 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000368778 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "req_readdir_latency": { 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 76407, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 532.218201213, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.006965568 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rename_latency": { 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 4781, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 38.366824887, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.008024853 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "req_renamesnap_latency": { 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.330 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "req_snapdiff_latency": { 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmdir_latency": { 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 30, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.219849666, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.007328322 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmsnap_latency": { 2024-09-21T01:29:49.331 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmxattr_latency": { 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setattr_latency": { 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 153, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 39.803946968, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.260156516 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setdirlayout_latency": { 2024-09-21T01:29:49.332 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setfilelock_latency": { 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 768, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1.766834143, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.002300565 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setlayout_latency": { 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setxattr_latency": { 2024-09-21T01:29:49.333 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "req_symlink_latency": { 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "req_unlink_latency": { 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 23155, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 233.261045507, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.010073895 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_revoke_eviction": 0, 2024-09-21T01:29:49.334 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition_throttle": 0, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getvxattr_latency": { 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessions": { 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "session_count": 2, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "session_add": 3, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "session_remove": 1, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_open": 2, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_stale": 0, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "total_load": 18126, 2024-09-21T01:29:49.335 INFO:teuthology.orchestra.run.smithi039.stdout: "average_load": 9063, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "avg_session_uptime": 1545, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "mdthresh_evicted": 0 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:29:49.336 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:29:49.337 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 960570, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 1179, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 264, 2024-09-21T01:29:49.338 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 3, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 9736, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 197, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 8472943, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 109240, 2024-09-21T01:29:49.339 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter": { 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_active": 0, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_laggy": 0, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send": 84017, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send_bytes": 675760060, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_resend": 27, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_reply": 84017, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "op_latency": { 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 84017, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1151.129882464, 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.013701154 2024-09-21T01:29:49.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "op_inflight": 0, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "oplen_avg": { 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 84017, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 86238 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "op": 84017, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "op_r": 1003, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "op_w": 83014, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "op_rmw": 0, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "op_pg": 0, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_stat": 0, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_create": 393, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_read": 962, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_write": 34939, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writefull": 572, 2024-09-21T01:29:49.341 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writesame": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_append": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_zero": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_truncate": 9, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_delete": 46046, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_mapext": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_sparse_read": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_clonerange": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_getxattr": 12, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_setxattr": 615, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_cmpxattr": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_rmxattr": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_resetxattrs": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_call": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_watch": 0, 2024-09-21T01:29:49.342 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_notify": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls_filter": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_other": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_active": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_send": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_resend": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_ping": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_active": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_send": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_resend": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_active": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_send": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_resend": 0, 2024-09-21T01:29:49.343 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_active": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_send": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_resend": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "command_active": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "command_send": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "command_resend": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "map_epoch": 130, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "map_full": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "map_inc": 42, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_sessions": 12, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_open": 12, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_close": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_laggy": 0, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_wr": 1964, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_rd": 82, 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_del": 644 2024-09-21T01:29:49.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "oft": { 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_objs": 1, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_kv_pairs": 0, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_updates": 0, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_removes": 0 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "purge_queue": { 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops": 0, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops_high_water": 16, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing": 0, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_high_water": 8, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed_ops": 45838, 2024-09-21T01:29:49.345 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed": 22919, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_item_in_journal": 0 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 591077, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 168833033, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 591077, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 591077, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 168833033, 2024-09-21T01:29:49.346 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_bytes": { 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-09-21T01:29:49.347 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 84017, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 678269983, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 84017, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 678269983, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_ops": { 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 1024, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-09-21T01:29:49.348 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 84017, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 84017, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 84017, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 84017, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle": { 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2024-09-21T01:29:49.349 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 22919, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 2510579, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 22919, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 962, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 2510579, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.350 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle-0x564a4510b450": { 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 129789, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 668512836, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 129789, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 33977, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 668512836, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:49.351 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:49.352 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:49.352 INFO:teuthology.orchestra.run.smithi039.stdout: "trackedop": { 2024-09-21T01:29:49.352 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_ops_count": 3008 2024-09-21T01:29:49.352 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:49.352 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:29:49.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:48 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:29:50.068 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:49 smithi039 ceph-mon[30415]: pgmap v1218: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-09-21T01:29:50.109 INFO:tasks.check_counter:Daemon mds.a mds.exported=49 2024-09-21T01:29:50.110 INFO:tasks.check_counter:Daemon mds.a mds.imported=15 2024-09-21T01:29:50.110 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=6 2024-09-21T01:29:50.110 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-09-21T01:29:50.110 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T01:29:50.110 DEBUG:tasks.check_counter:Getting stats from d 2024-09-21T01:29:50.110 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.d perf dump 2024-09-21T01:29:50.158 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:49 smithi150 ceph-mon[34945]: pgmap v1218: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-09-21T01:29:50.169 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:49 smithi160 ceph-mon[34696]: pgmap v1218: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-09-21T01:29:50.953 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:29:52.064 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:51 smithi150 ceph-mon[34945]: pgmap v1219: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 480 KiB/s wr, 55 op/s 2024-09-21T01:29:52.206 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:51 smithi039 ceph-mon[30415]: pgmap v1219: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 480 KiB/s wr, 55 op/s 2024-09-21T01:29:52.349 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:51 smithi160 ceph-mon[34696]: pgmap v1219: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 480 KiB/s wr, 55 op/s 2024-09-21T01:29:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:53 smithi150 ceph-mon[34945]: pgmap v1220: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 1.6 MiB/s wr, 49 op/s 2024-09-21T01:29:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:53 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:53 smithi160 ceph-mon[34696]: pgmap v1220: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 1.6 MiB/s wr, 49 op/s 2024-09-21T01:29:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:53 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:53 smithi039 ceph-mon[30415]: pgmap v1220: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 1.6 MiB/s wr, 49 op/s 2024-09-21T01:29:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:54.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:53 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:29:56.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:55 smithi150 ceph-mon[34945]: pgmap v1221: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.5 MiB/s wr, 86 op/s 2024-09-21T01:29:56.319 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:55 smithi039 ceph-mon[30415]: pgmap v1221: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.5 MiB/s wr, 86 op/s 2024-09-21T01:29:56.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:55 smithi160 ceph-mon[34696]: pgmap v1221: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.5 MiB/s wr, 86 op/s 2024-09-21T01:29:58.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:57 smithi150 ceph-mon[34945]: pgmap v1222: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 3.7 MiB/s wr, 93 op/s 2024-09-21T01:29:58.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:57 smithi160 ceph-mon[34696]: pgmap v1222: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 3.7 MiB/s wr, 93 op/s 2024-09-21T01:29:58.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:57 smithi039 ceph-mon[30415]: pgmap v1222: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 3.7 MiB/s wr, 93 op/s 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 0, 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 2, 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 0, 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 0, 2024-09-21T01:29:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 0, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.003634472, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.000000000, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.000638060, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:59.017 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 6, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 365, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 3264, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 225072, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-09-21T01:29:59.018 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.051893798, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.042217112, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.005368089, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 364, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.050066545, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000137545 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.019 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 225072 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 479, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 474, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 911488, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 294032, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 2, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 1, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.085918813, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.040770248, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.056645113, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:29:59.020 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 473, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.066536853, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000140669 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 911488, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 294032 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.021 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:29:59.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:29:59.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:29:59.024 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 149735, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 30, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-09-21T01:29:59.025 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 968, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 481, 2024-09-21T01:29:59.026 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 843716, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 481, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 481, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 843716, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:29:59.027 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:29:59.860 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-21T01:29:59.861 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-21T01:29:59.861 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-09-21T01:29:59.861 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-09-21T01:29:59.861 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-09-21T01:29:59.861 DEBUG:tasks.check_counter:Getting stats from g 2024-09-21T01:29:59.861 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.g perf dump 2024-09-21T01:30:00.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:29:59 smithi039 ceph-mon[30415]: pgmap v1223: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 4.3 MiB/s wr, 155 op/s 2024-09-21T01:30:00.279 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:00.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:29:59 smithi150 ceph-mon[34945]: pgmap v1223: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 4.3 MiB/s wr, 155 op/s 2024-09-21T01:30:00.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:29:59 smithi160 ceph-mon[34696]: pgmap v1223: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 4.3 MiB/s wr, 155 op/s 2024-09-21T01:30:01.236 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:00 smithi039 ceph-mon[30415]: overall HEALTH_OK 2024-09-21T01:30:01.271 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:30:01.271 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:01.271 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 475, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 470, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 910400, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 291472, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 0, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.101121524, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.057291142, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.049614009, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 469, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.032018214, 2024-09-21T01:30:01.272 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000068269 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 910400, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 291472 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 2, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 360, 2024-09-21T01:30:01.273 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 320, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 224096, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.047621503, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.040844551, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.002211940, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 358, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.034073177, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000095176 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 224096 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 4, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 4, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 2944, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 448, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 2, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 1, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.003720239, 2024-09-21T01:30:01.275 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.000067781, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.003605451, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.006031803, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.002010601 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T01:30:01.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:01.277 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:01.278 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 148704, 2024-09-21T01:30:01.279 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 29, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 968, 2024-09-21T01:30:01.280 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 477, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 843196, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 477, 2024-09-21T01:30:01.281 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 477, 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 843196, 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:30:01.282 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:30:01.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:00 smithi150 ceph-mon[34945]: overall HEALTH_OK 2024-09-21T01:30:01.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:00 smithi160 ceph-mon[34696]: overall HEALTH_OK 2024-09-21T01:30:02.294 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: pgmap v1224: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 4.4 MiB/s wr, 152 op/s 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:30:02.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:01 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: pgmap v1224: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 4.4 MiB/s wr, 152 op/s 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:30:02.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:01 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.333 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-09-21T01:30:02.333 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-09-21T01:30:02.333 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-09-21T01:30:02.333 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-09-21T01:30:02.333 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-09-21T01:30:02.334 DEBUG:tasks.check_counter:Getting stats from j 2024-09-21T01:30:02.334 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.j perf dump 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: pgmap v1224: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 4.4 MiB/s wr, 152 op/s 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:30:02.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:02 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' 2024-09-21T01:30:02.716 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:03.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:03 smithi039 ceph-mon[30415]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:30:03.131 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:03 smithi039 ceph-mon[30415]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:30:03.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:03 smithi150 ceph-mon[34945]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:30:03.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:03 smithi150 ceph-mon[34945]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:30:03.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:03 smithi160 ceph-mon[34696]: Adjusting osd_memory_target on smithi039 to 176.4M 2024-09-21T01:30:03.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:03 smithi160 ceph-mon[34696]: Unable to set osd_memory_target on smithi039 to 185042636: error parsing value: Value '185042636' is below minimum 939524096 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 4, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 4, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 2944, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 448, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 0, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.005868354, 2024-09-21T01:30:03.612 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.000073196, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.003297859, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.005697285, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.001899095 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T01:30:03.613 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 3, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 366, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 480, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 305696, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 5, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.053297099, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.045275876, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.002963626, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 364, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.036161237, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000099344 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 305696 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 482, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 477, 2024-09-21T01:30:03.615 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 912928, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 295648, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 2, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 1, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.103202174, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.058815744, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.053014910, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 476, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.039608787, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000083211 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:03.616 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 912928, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 295648 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 0, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:03.617 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:03.618 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:03.619 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 148704, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 29, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:03.620 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 968, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-09-21T01:30:03.621 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 485, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 844770, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 485, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 485, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 844770, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-09-21T01:30:03.622 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:30:03.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T01:30:03.623 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T01:30:04.265 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:04 smithi160 ceph-mon[34696]: pgmap v1225: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 4.5 MiB/s wr, 156 op/s 2024-09-21T01:30:04.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:04 smithi150 ceph-mon[34945]: pgmap v1225: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 4.5 MiB/s wr, 156 op/s 2024-09-21T01:30:04.343 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:04 smithi039 ceph-mon[30415]: pgmap v1225: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 4.5 MiB/s wr, 156 op/s 2024-09-21T01:30:04.381 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-21T01:30:04.382 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-21T01:30:04.382 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-09-21T01:30:04.382 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-09-21T01:30:04.382 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-09-21T01:30:04.382 DEBUG:tasks.check_counter:Getting stats from b 2024-09-21T01:30:04.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.b perf dump 2024-09-21T01:30:04.706 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T01:30:05.529 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-21T01:30:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 245080, 2024-09-21T01:30:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 256377, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 86946822, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 572060722, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 10, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 5, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 18.296605760, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 8.975300541, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 8.951037407, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 1.735956072, 2024-09-21T01:30:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 256377, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 10.694877754, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000041715 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 400196, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.071588382, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000178 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 2034544, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 609488 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 402872, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 594150, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 129871511, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 821952984, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 118, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 6, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 30.015469243, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 17.954817126, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 13.548089707, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 1.858881279, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 594148, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 24.504124559, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000041242 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 431863, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.128253966, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000296 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 55776, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 862960 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 193071, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 208722, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 68517945, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 476077249, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 11, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 6, 2024-09-21T01:30:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 15.318091840, 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 7.779443741, 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 6.761665115, 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 1.668456495, 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 208720, 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 10.373812738, 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000049702 2024-09-21T01:30:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 287770, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.050091208, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000174 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 1, 2024-09-21T01:30:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-MDSRank": { 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 118478, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 133.510717634, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.001126881 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-PurgeQueue": { 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 66098, 2024-09-21T01:30:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 8.961249553, 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000135575 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": { 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "request": 253219, 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "reply": 240905, 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "reply_latency": { 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 240905, 2024-09-21T01:30:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 1787.978088303, 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.007421921 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_reply": 0, 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "forward": 12314, 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_complete": 13, 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_keys": 0, 2024-09-21T01:30:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_commit": 203, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_split": 13, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_merge": 8, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes": 3277, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_top": 1373, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_bottom": 1893, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pin_tail": 11, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pinned": 1315, 2024-09-21T01:30:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_expired": 3293, 2024-09-21T01:30:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_with_caps": 2, 2024-09-21T01:30:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": 2, 2024-09-21T01:30:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "subtrees": 4, 2024-09-21T01:30:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse": 336432, 2024-09-21T01:30:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_hit": 229600, 2024-09-21T01:30:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_forward": 12260, 2024-09-21T01:30:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_discover": 168, 2024-09-21T01:30:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_dir_fetch": 2, 2024-09-21T01:30:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_remote_ino": 0, 2024-09-21T01:30:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_lock": 979, 2024-09-21T01:30:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "load_cent": 31, 2024-09-21T01:30:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "q": 0, 2024-09-21T01:30:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "exported": 24, 2024-09-21T01:30:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "exported_inodes": 2149, 2024-09-21T01:30:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "imported": 37, 2024-09-21T01:30:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "imported_inodes": 724, 2024-09-21T01:30:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_dir_fetch": 0, 2024-09-21T01:30:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_backtrace_fetch": 0, 2024-09-21T01:30:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_peer_discover": 0, 2024-09-21T01:30:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rfiles": 4, 2024-09-21T01:30:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rbytes": 234, 2024-09-21T01:30:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rsnaps": 6, 2024-09-21T01:30:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T01:30:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_set_tag": 0, 2024-09-21T01:30:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T01:30:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_inotable_repaired": 0, 2024-09-21T01:30:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_inodes": 0, 2024-09-21T01:30:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T01:30:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T01:30:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_file_inodes": 0, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_inode_file_caps": 41, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_revoke": 30943, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_grant": 153499, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flushsnap_ack": 6, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T01:30:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps": 89683, 2024-09-21T01:30:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps_dirty": 28194, 2024-09-21T01:30:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_cap_release": 6403, 2024-09-21T01:30:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "process_request_cap_release": 155667 2024-09-21T01:30:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache": { 2024-09-21T01:30:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update": 10, 2024-09-21T01:30:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update_receipt": 18, 2024-09-21T01:30:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_try_discover": 12, 2024-09-21T01:30:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_send_discover": 175, 2024-09-21T01:30:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_handle_discover": 178, 2024-09-21T01:30:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays": 43, 2024-09-21T01:30:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_delayed": 0, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_enqueuing": 0, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_created": 35580, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_enqueued": 35537, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_reintegrated": 0, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_migrated": 0, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_processing": 0, 2024-09-21T01:30:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_enqueued": 0, 2024-09-21T01:30:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_prioritized": 0, 2024-09-21T01:30:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_started": 0, 2024-09-21T01:30:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_completed": 0, 2024-09-21T01:30:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_quiesce_path": 25, 2024-09-21T01:30:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_quiesce_inode": 4413, 2024-09-21T01:30:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T01:30:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_exportdir": 26, 2024-09-21T01:30:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_flush": 0, 2024-09-21T01:30:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragmentdir": 21, 2024-09-21T01:30:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragstats": 0, 2024-09-21T01:30:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_inodestats": 0, 2024-09-21T01:30:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "uninline_started": 0, 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "uninline_succeeded": 0, 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "uninline_write_failed": 0 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log": { 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "evlrg": 0, 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "evadd": 185914, 2024-09-21T01:30:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "evex": 130282, 2024-09-21T01:30:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "evtrm": 122901, 2024-09-21T01:30:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "ev": 63013, 2024-09-21T01:30:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "evexg": 0, 2024-09-21T01:30:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "evexd": 7381, 2024-09-21T01:30:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "segadd": 347, 2024-09-21T01:30:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "segex": 219, 2024-09-21T01:30:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "segtrm": 205, 2024-09-21T01:30:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "seg": 142, 2024-09-21T01:30:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "segmjr": 6, 2024-09-21T01:30:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "segexg": 0, 2024-09-21T01:30:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "segexd": 14, 2024-09-21T01:30:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "expos": 722355118, 2024-09-21T01:30:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "wrpos": 1293648927, 2024-09-21T01:30:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "rdpos": 0, 2024-09-21T01:30:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "jlat": { 2024-09-21T01:30:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 65274, 2024-09-21T01:30:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 484.189979350, 2024-09-21T01:30:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.007417807 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "replayed": 0 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_mem": { 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "ino": 2408, 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "ino+": 39929, 2024-09-21T01:30:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "ino-": 37521, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dir": 124, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dir+": 395, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dir-": 271, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dn": 3271, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dn+": 75349, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dn-": 72078, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "cap": 2, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "cap+": 38823, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "cap-": 38821, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "rss": 159764, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "heap": 378421248 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_server": { 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_client_request": 357940, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_server_request": 15567, 2024-09-21T01:30:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_request": 253219, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_session": 4127, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_peer_request": 29943, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "req_create_latency": { 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 36166, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 468.036421660, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.012941337 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getattr_latency": { 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 17724, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 148.203973195, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.008361767 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getfilelock_latency": { 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "req_link_latency": { 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookup_latency": { 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 48345, 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 136.825309336, 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002830185 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookuphash_latency": { 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupino_latency": { 2024-09-21T01:30:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupname_latency": { 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupparent_latency": { 2024-09-21T01:30:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupsnap_latency": { 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lssnap_latency": { 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 27, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.008215289, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000304269 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mkdir_latency": { 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 119, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 16.991954408, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.142789532 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mknod_latency": { 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mksnap_latency": { 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 27, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 29.343740302, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 1.086805196 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "req_open_latency": { 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 15, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.021530629, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.001435375 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "req_readdir_latency": { 2024-09-21T01:30:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 94662, 2024-09-21T01:30:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 582.629002749, 2024-09-21T01:30:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.006154835 2024-09-21T01:30:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rename_latency": { 2024-09-21T01:30:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 7082, 2024-09-21T01:30:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 76.289127337, 2024-09-21T01:30:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.010772257 2024-09-21T01:30:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "req_renamesnap_latency": { 2024-09-21T01:30:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "req_snapdiff_latency": { 2024-09-21T01:30:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmdir_latency": { 2024-09-21T01:30:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 54, 2024-09-21T01:30:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.807399771, 2024-09-21T01:30:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.014951847 2024-09-21T01:30:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmsnap_latency": { 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 21, 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 3.755074106, 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.178813052 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmxattr_latency": { 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setattr_latency": { 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 192, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 45.974869200, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.239452443 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setdirlayout_latency": { 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setfilelock_latency": { 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 924, 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 2.031220239, 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002198290 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setlayout_latency": { 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setxattr_latency": { 2024-09-21T01:30:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 10, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.025269300, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002526930 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "req_symlink_latency": { 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "req_unlink_latency": { 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 35537, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 277.034980782, 2024-09-21T01:30:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.007795677 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_revoke_eviction": 0, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_acquisition_throttle": 0, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getvxattr_latency": { 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sessions": { 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "session_count": 2, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "session_add": 4, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "session_remove": 2, 2024-09-21T01:30:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_open": 2, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_stale": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "total_load": 14422, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "average_load": 7211, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "avg_session_uptime": 1572, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "mdthresh_evicted": 0 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 766497, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 1114, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 264, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 3, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 9736, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 197, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 8806537, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 87297, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter": { 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "op_active": 0, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "op_laggy": 0, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send": 142077, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send_bytes": 1299913544, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "op_resend": 37, 2024-09-21T01:30:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "op_reply": 142077, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op_latency": { 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 142077, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 1414.261881501, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.009954193 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op_inflight": 0, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "oplen_avg": { 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 142077, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 144663 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op": 142077, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op_r": 1150, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op_w": 140927, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op_rmw": 0, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "op_pg": 0, 2024-09-21T01:30:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_stat": 32, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_create": 462, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_read": 1137, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_write": 66411, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writefull": 699, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writesame": 0, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_append": 0, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_zero": 0, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_truncate": 13, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_delete": 72122, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_mapext": 0, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_sparse_read": 0, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_clonerange": 0, 2024-09-21T01:30:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_getxattr": 5, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_setxattr": 734, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_cmpxattr": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_rmxattr": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_resetxattrs": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_call": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_watch": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_notify": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls_filter": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_other": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_active": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_send": 0, 2024-09-21T01:30:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_resend": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_ping": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_active": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_send": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_resend": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_active": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_send": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_resend": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_active": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_send": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_resend": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "command_active": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "command_send": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "command_resend": 0, 2024-09-21T01:30:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "map_epoch": 130, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "map_full": 0, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "map_inc": 51, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_sessions": 12, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_open": 12, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_close": 0, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_laggy": 0, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_wr": 2264, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_rd": 26, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_del": 758 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "oft": { 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_objs": 1, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_kv_pairs": 0, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_updates": 0, 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_removes": 0 2024-09-21T01:30:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "purge_queue": { 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops": 0, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops_high_water": 16, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing": 0, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_high_water": 8, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed_ops": 71074, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed": 35537, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_item_in_journal": 0 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 841019, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 215222643, 2024-09-21T01:30:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 841019, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 841019, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 215222643, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_bytes": { 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-09-21T01:30:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 142077, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 1303801949, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 142077, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1303801949, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_ops": { 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 1024, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 142077, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 142077, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 142077, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 142077, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle": { 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 35537, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 3888613, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 35537, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 1137, 2024-09-21T01:30:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 3888613, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle-0x5648c20dd450": { 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 60, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 185914, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 1289454623, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 185914, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 65274, 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1289454563, 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "trackedop": { 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_ops_count": 3372 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:05.590 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-21T01:30:06.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:06 smithi150 ceph-mon[34945]: pgmap v1226: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-09-21T01:30:06.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:06 smithi160 ceph-mon[34696]: pgmap v1226: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-09-21T01:30:06.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:06 smithi039 ceph-mon[30415]: pgmap v1226: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-09-21T01:30:06.626 INFO:tasks.check_counter:Daemon mds.b mds.exported=24 2024-09-21T01:30:06.626 INFO:tasks.check_counter:Daemon mds.b mds.imported=37 2024-09-21T01:30:06.626 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-09-21T01:30:06.626 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=27 2024-09-21T01:30:06.626 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=21 2024-09-21T01:30:06.626 DEBUG:tasks.check_counter:Getting stats from e 2024-09-21T01:30:06.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.e perf dump 2024-09-21T01:30:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T01:30:07.556 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 0, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 2, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 0, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 0, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.001127952, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000000000, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.000236081, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:07.557 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:07.558 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 484, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 477, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 914704, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 295120, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.076349045, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.038010133, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.048993896, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 477, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.028755542, 2024-09-21T01:30:07.559 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000060284 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 914704, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 295120 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 2, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 364, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 320, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 224800, 2024-09-21T01:30:07.560 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 2, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.028141036, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.023122741, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.001304152, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 362, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.022178549, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000061266 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.561 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 224800 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:07.562 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:07.563 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:07.564 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:07.565 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:07.566 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-09-21T01:30:07.566 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:07.566 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:07.566 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:07.566 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 148761, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 31, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 968, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-09-21T01:30:07.568 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 482, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 843846, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 482, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 482, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 843846, 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:07.569 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:07.570 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:07.570 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:07.570 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:07.570 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:07.570 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-21T01:30:08.297 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:08 smithi150 ceph-mon[34945]: pgmap v1227: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-09-21T01:30:08.326 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-09-21T01:30:08.326 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-09-21T01:30:08.326 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-09-21T01:30:08.326 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-09-21T01:30:08.326 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-09-21T01:30:08.326 DEBUG:tasks.check_counter:Getting stats from h 2024-09-21T01:30:08.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.h perf dump 2024-09-21T01:30:08.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:08 smithi160 ceph-mon[34696]: pgmap v1227: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-09-21T01:30:08.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:08 smithi039 ceph-mon[30415]: pgmap v1227: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-09-21T01:30:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 3274, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 3224, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 186540413, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 1161622, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 7, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 3, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.486131039, 2024-09-21T01:30:09.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.177818115, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.220421588, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.076232005, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 3224, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.142366903, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000044158 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1153840, 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 302704 2024-09-21T01:30:09.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 1324, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 1685, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 193289922, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 1227978, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 8, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 6, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.255519906, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.068164011, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.142531325, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.033188514, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 1683, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.061184842, 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000036354 2024-09-21T01:30:09.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 807520 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 2582, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 2584, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 294597271, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 821083, 2024-09-21T01:30:09.110 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 7, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 5, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.425690043, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.081885906, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.223318665, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.104863347, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2582, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.144608564, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000056006 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 1, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-MDSRank": { 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 10806, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.275416364, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000025487 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.112 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-PurgeQueue": { 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": { 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "request": 0, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "reply": 0, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "reply_latency": { 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_reply": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "forward": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_complete": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_keys": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_commit": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_split": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_merge": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes": 25792, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_top": 5371, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_bottom": 20421, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pin_tail": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pinned": 18118, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_expired": 0, 2024-09-21T01:30:09.114 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_with_caps": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "subtrees": 1, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_hit": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_forward": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_discover": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_dir_fetch": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_remote_ino": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_lock": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "load_cent": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "q": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "exported": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "exported_inodes": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "imported": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "imported_inodes": 0, 2024-09-21T01:30:09.115 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_dir_fetch": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_backtrace_fetch": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_peer_discover": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rfiles": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rbytes": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rsnaps": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_set_tag": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_inotable_repaired": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_inodes": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_file_inodes": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_inode_file_caps": 0, 2024-09-21T01:30:09.116 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_grant": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps_dirty": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_cap_release": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "process_request_cap_release": 0 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache": { 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update_receipt": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_try_discover": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_send_discover": 0, 2024-09-21T01:30:09.117 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_handle_discover": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_delayed": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_enqueuing": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_created": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_enqueued": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_reintegrated": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_migrated": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_processing": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_enqueued": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_prioritized": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_started": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_completed": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_quiesce_path": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_quiesce_inode": 0, 2024-09-21T01:30:09.118 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_exportdir": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_flush": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragmentdir": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragstats": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_inodestats": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "uninline_started": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "uninline_succeeded": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "uninline_write_failed": 0 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log": { 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "evlrg": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "evadd": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "evex": 0, 2024-09-21T01:30:09.119 INFO:teuthology.orchestra.run.smithi150.stdout: "evtrm": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "ev": 130264, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "evexg": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "evexd": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "segadd": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "segex": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "segtrm": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "seg": 136, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "segmjr": 12, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "segexg": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "segexd": 0, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "expos": 143699901, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "wrpos": 675562985, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "rdpos": 675548126, 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "jlat": { 2024-09-21T01:30:09.120 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "replayed": 130264 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_mem": { 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "ino": 2188, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "ino+": 26299, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "ino-": 24111, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "dir": 83, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "dir+": 158, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "dir-": 75, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "dn": 25792, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "dn+": 26912, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "dn-": 1120, 2024-09-21T01:30:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: "cap": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "cap+": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "cap-": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "rss": 152572, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "heap": 366014464 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_server": { 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_client_request": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_server_request": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_request": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_session": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_peer_request": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "req_create_latency": { 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getattr_latency": { 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getfilelock_latency": { 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "req_link_latency": { 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookup_latency": { 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookuphash_latency": { 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupino_latency": { 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupname_latency": { 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupparent_latency": { 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupsnap_latency": { 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.126 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lssnap_latency": { 2024-09-21T01:30:09.126 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.126 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.126 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.126 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mkdir_latency": { 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mknod_latency": { 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mksnap_latency": { 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "req_open_latency": { 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "req_readdir_latency": { 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rename_latency": { 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "req_renamesnap_latency": { 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "req_snapdiff_latency": { 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmdir_latency": { 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmsnap_latency": { 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmxattr_latency": { 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setattr_latency": { 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.130 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setdirlayout_latency": { 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setfilelock_latency": { 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setlayout_latency": { 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setxattr_latency": { 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "req_symlink_latency": { 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "req_unlink_latency": { 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_revoke_eviction": 0, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_acquisition_throttle": 0, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getvxattr_latency": { 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sessions": { 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "session_count": 2, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "session_add": 3, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "session_remove": 1, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_open": 2, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_stale": 0, 2024-09-21T01:30:09.133 INFO:teuthology.orchestra.run.smithi150.stdout: "total_load": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "average_load": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "avg_session_uptime": 3199, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "mdthresh_evicted": 0 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:09.134 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 332672, 2024-09-21T01:30:09.136 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 2541, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 9736, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 197, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-09-21T01:30:09.137 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 23349899, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 182441, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter": { 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_active": 0, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_laggy": 0, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send": 6605, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send_bytes": 21, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_resend": 3, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_reply": 6605, 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "op_latency": { 2024-09-21T01:30:09.138 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 6605, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 16.954675353, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002566945 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "op_inflight": 0, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "oplen_avg": { 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 6605, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 6606 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "op": 6605, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "op_r": 6605, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "op_w": 0, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "op_rmw": 0, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "op_pg": 0, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_stat": 3869, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_create": 0, 2024-09-21T01:30:09.139 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_read": 2734, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_write": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writefull": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writesame": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_append": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_zero": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_truncate": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_delete": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_mapext": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_sparse_read": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_clonerange": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_getxattr": 1, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_setxattr": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_cmpxattr": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_rmxattr": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_resetxattrs": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_call": 0, 2024-09-21T01:30:09.140 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_watch": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_notify": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls_filter": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_other": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_active": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_send": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_resend": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_ping": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_active": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_send": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_resend": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_active": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_send": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_resend": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_active": 0, 2024-09-21T01:30:09.141 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_send": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_resend": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "command_active": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "command_send": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "command_resend": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "map_epoch": 130, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "map_full": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "map_inc": 42, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_sessions": 11, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_open": 12, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_close": 1, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_laggy": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_wr": 0, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_rd": 2, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_del": 0 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "oft": { 2024-09-21T01:30:09.142 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_objs": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_kv_pairs": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_updates": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_removes": 0 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "purge_queue": { 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_high_water": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed_ops": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed": 0, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_item_in_journal": 0 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:09.143 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 7176, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 673749775, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 7176, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 7176, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 673749775, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.144 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_bytes": { 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 6605, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 671452294, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 6605, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 671452294, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_ops": { 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 1024, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 6605, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 6605, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 6605, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 6605, 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:09.146 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle": { 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:09.147 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle-0x55e2b0917520": { 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 0, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 0, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "trackedop": { 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_ops_count": 0 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:09.149 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-21T01:30:09.896 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-09-21T01:30:09.896 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-09-21T01:30:09.896 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-09-21T01:30:09.896 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-09-21T01:30:09.896 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T01:30:09.896 DEBUG:tasks.check_counter:Getting stats from k 2024-09-21T01:30:09.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.k perf dump 2024-09-21T01:30:10.225 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T01:30:10.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:10 smithi150 ceph-mon[34945]: pgmap v1228: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 849 KiB/s wr, 95 op/s 2024-09-21T01:30:10.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:10 smithi160 ceph-mon[34696]: pgmap v1228: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 849 KiB/s wr, 95 op/s 2024-09-21T01:30:10.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:10 smithi039 ceph-mon[30415]: pgmap v1228: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 849 KiB/s wr, 95 op/s 2024-09-21T01:30:10.916 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-21T01:30:10.916 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:10.916 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 485, 2024-09-21T01:30:10.916 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 480, 2024-09-21T01:30:10.916 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 913744, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 297568, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.076034051, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.038900543, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.046478542, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 479, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.023791248, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000049668 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:10.917 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 913744, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 297568 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 3, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 368, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 480, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 306048, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 5, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 2, 2024-09-21T01:30:10.918 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.027966677, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.022390519, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.001848541, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 366, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.072708807, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000198657 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.919 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 306048 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 4, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 4, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 2944, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 448, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.002190817, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000046850, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.002140736, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:10.920 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 3, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.003868081, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.001289360 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-09-21T01:30:10.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:10.922 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:10.924 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 147920, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 27, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:10.925 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 968, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 488, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 845160, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 488, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 488, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 845160, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-21T01:30:10.927 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-21T01:30:11.288 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:11 smithi150 ceph-mon[34945]: osdmap e131: 12 total, 12 up, 12 in 2024-09-21T01:30:11.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:11 smithi160 ceph-mon[34696]: osdmap e131: 12 total, 12 up, 12 in 2024-09-21T01:30:11.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:11 smithi039 ceph-mon[30415]: osdmap e131: 12 total, 12 up, 12 in 2024-09-21T01:30:12.025 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-09-21T01:30:12.025 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-09-21T01:30:12.025 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-09-21T01:30:12.025 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-09-21T01:30:12.025 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-09-21T01:30:12.025 DEBUG:tasks.check_counter:Getting stats from c 2024-09-21T01:30:12.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.c perf dump 2024-09-21T01:30:12.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:12 smithi150 ceph-mon[34945]: pgmap v1230: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 119 KiB/s wr, 36 op/s 2024-09-21T01:30:12.342 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:30:12.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:12 smithi160 ceph-mon[34696]: pgmap v1230: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 119 KiB/s wr, 36 op/s 2024-09-21T01:30:12.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:12 smithi039 ceph-mon[30415]: pgmap v1230: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 119 KiB/s wr, 36 op/s 2024-09-21T01:30:13.979 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T01:30:13.979 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:13.979 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 1574, 2024-09-21T01:30:13.979 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 1526, 2024-09-21T01:30:13.979 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 236305729, 2024-09-21T01:30:13.979 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 621868, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 7, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 3, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.315538526, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.076056821, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.217795004, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.025771067, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 1525, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.057981461, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000038020 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.980 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 1152640, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 303232 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 4603, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 4963, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 601182755, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 2180017, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 7, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 6, 2024-09-21T01:30:13.981 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.812830990, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.213531101, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.431438086, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.131187474, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 4962, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.187443324, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000037775 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 717792 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 2033, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 2035, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 455373401, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 646510, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 6, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 5, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.407362117, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.083107365, 2024-09-21T01:30:13.983 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.257750308, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.052179772, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 2033, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.071374331, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000035107 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T01:30:13.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "cct": { 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "total_workers": 1, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "unhealthy_workers": 0 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "finisher-MDSRank": { 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "queue_len": 0, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "complete_latency": { 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 11970, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.365111129, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000030502 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "finisher-PurgeQueue": { 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "queue_len": 0, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "complete_latency": { 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.985 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "mds": { 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "request": 0, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "reply": 0, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "reply_latency": { 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "slow_reply": 0, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "forward": 0, 2024-09-21T01:30:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_fetch_complete": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_fetch_keys": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_commit": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_split": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_merge": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes": 38484, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_top": 16851, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_bottom": 21633, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_pin_tail": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_pinned": 14411, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_expired": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_with_caps": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "caps": 0, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "subtrees": 2, 2024-09-21T01:30:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_hit": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_forward": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_discover": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_dir_fetch": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_remote_ino": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_lock": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "load_cent": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "q": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "exported": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "exported_inodes": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "imported": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "imported_inodes": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_dir_fetch": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_backtrace_fetch": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_peer_discover": 0, 2024-09-21T01:30:13.988 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rfiles": 4, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rbytes": 234, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rsnaps": 6, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_set_tag": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_inotable_repaired": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dir_inodes": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_file_inodes": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_inode_file_caps": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_grant": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T01:30:13.989 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_caps": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_caps_dirty": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_cap_release": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "process_request_cap_release": 0 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_cache": { 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_update": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_update_receipt": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_try_discover": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_send_discover": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_handle_discover": 0, 2024-09-21T01:30:13.990 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays_delayed": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays_enqueuing": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_created": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_enqueued": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_reintegrated": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_migrated": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_processing": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_enqueued": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_prioritized": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "recovery_started": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "recovery_completed": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_quiesce_path": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_quiesce_inode": 0, 2024-09-21T01:30:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_exportdir": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_flush": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_fragmentdir": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_fragstats": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_inodestats": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_started": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_succeeded": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_write_failed": 0 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_log": { 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "evlrg": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "evadd": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "evex": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "evtrm": 0, 2024-09-21T01:30:13.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ev": 185915, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "evexg": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "evexd": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "segadd": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "segex": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "segtrm": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "seg": 142, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "segmjr": 17, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "segexg": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "segexd": 0, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "expos": 722355118, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "wrpos": 1293663313, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "rdpos": 1293648927, 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "jlat": { 2024-09-21T01:30:13.993 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "replayed": 185915 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_mem": { 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "ino": 2387, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "ino+": 39626, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "ino-": 37239, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "dir": 103, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "dir+": 285, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "dir-": 182, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "dn": 38484, 2024-09-21T01:30:13.994 INFO:teuthology.orchestra.run.smithi160.stdout: "dn+": 40620, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "dn-": 2136, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "cap": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "cap+": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "cap-": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "rss": 161632, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "heap": 372740096 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_server": { 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "dispatch_client_request": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "dispatch_server_request": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_request": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_session": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_peer_request": 0, 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "req_create_latency": { 2024-09-21T01:30:13.995 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getattr_latency": { 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getfilelock_latency": { 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.996 INFO:teuthology.orchestra.run.smithi160.stdout: "req_link_latency": { 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookup_latency": { 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookuphash_latency": { 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupino_latency": { 2024-09-21T01:30:13.997 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupname_latency": { 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupparent_latency": { 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.998 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupsnap_latency": { 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lssnap_latency": { 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:13.999 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mkdir_latency": { 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mknod_latency": { 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mksnap_latency": { 2024-09-21T01:30:14.000 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "req_open_latency": { 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "req_readdir_latency": { 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.001 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rename_latency": { 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "req_renamesnap_latency": { 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "req_snapdiff_latency": { 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.002 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmdir_latency": { 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmsnap_latency": { 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmxattr_latency": { 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.003 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setattr_latency": { 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setdirlayout_latency": { 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setfilelock_latency": { 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.004 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setlayout_latency": { 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setxattr_latency": { 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "req_symlink_latency": { 2024-09-21T01:30:14.005 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "req_unlink_latency": { 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "cap_revoke_eviction": 0, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "cap_acquisition_throttle": 0, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getvxattr_latency": { 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.006 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_sessions": { 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "session_count": 2, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "session_add": 4, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "session_remove": 2, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "sessions_open": 2, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "sessions_stale": 0, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "total_load": 0, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "average_load": 0, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "avg_session_uptime": 3205, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "mdthresh_evicted": 0 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.007 INFO:teuthology.orchestra.run.smithi160.stdout: "mempool": { 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:14.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:14.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_items": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_bytes": 343149, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_items": 2862, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_bytes": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_items": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:14.010 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_bytes": 9736, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_items": 197, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_items": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_bytes": 32695356, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_items": 241537, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_items": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_items": 0 2024-09-21T01:30:14.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "objecter": { 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_active": 0, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_laggy": 0, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_send": 7633, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_send_bytes": 26, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_resend": 4, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_reply": 7633, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_latency": { 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 7633, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 19.121717466, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.002505137 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "op_inflight": 0, 2024-09-21T01:30:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "oplen_avg": { 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 7633, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 7634 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "op": 7633, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "op_r": 7633, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "op_w": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "op_rmw": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "op_pg": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_stat": 4571, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_create": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_read": 3059, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_write": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_writefull": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_writesame": 0, 2024-09-21T01:30:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_append": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_zero": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_truncate": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_delete": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_mapext": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_sparse_read": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_clonerange": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_getxattr": 2, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_setxattr": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_cmpxattr": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_rmxattr": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_resetxattrs": 0, 2024-09-21T01:30:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_call": 0, 2024-09-21T01:30:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_watch": 0, 2024-09-21T01:30:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_notify": 0, 2024-09-21T01:30:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T01:30:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_pgls": 0, 2024-09-21T01:30:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_pgls_filter": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_other": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_active": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_send": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_resend": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_ping": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_active": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_send": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_resend": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_active": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_send": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_resend": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_active": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_send": 0, 2024-09-21T01:30:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_resend": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "command_active": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "command_send": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "command_resend": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "map_epoch": 131, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "map_full": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "map_inc": 43, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_sessions": 11, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_session_open": 11, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_session_close": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_laggy": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_wr": 0, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_rd": 2, 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_del": 0 2024-09-21T01:30:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "oft": { 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_objs": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_kv_pairs": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_updates": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_removes": 0 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "purge_queue": { 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_ops": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_high_water": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executed_ops": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executed": 0, 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_item_in_journal": 0 2024-09-21T01:30:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 8206, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 1292091249, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 8206, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 8206, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 1292091249, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-objecter_bytes": { 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 7633, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 1289615123, 2024-09-21T01:30:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 7633, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 1289615123, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-objecter_ops": { 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 1024, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 7633, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 7633, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 7633, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 7633, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-write_buf_throttle": { 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 3758096384, 2024-09-21T01:30:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-write_buf_throttle-0x563edd2195f0": { 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 3758096384, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "trackedop": { 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "slow_ops_count": 0 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:14.025 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T01:30:14.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:14 smithi150 ceph-mon[34945]: pgmap v1231: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 73 KiB/s wr, 31 op/s 2024-09-21T01:30:14.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:14 smithi160 ceph-mon[34696]: pgmap v1231: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 73 KiB/s wr, 31 op/s 2024-09-21T01:30:14.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:14 smithi039 ceph-mon[30415]: pgmap v1231: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 73 KiB/s wr, 31 op/s 2024-09-21T01:30:14.964 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-21T01:30:14.964 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-21T01:30:14.964 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-09-21T01:30:14.964 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-09-21T01:30:14.964 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T01:30:14.964 DEBUG:tasks.check_counter:Getting stats from f 2024-09-21T01:30:14.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.f perf dump 2024-09-21T01:30:15.216 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:30:15.730 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T01:30:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 1937, 2024-09-21T01:30:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 1937, 2024-09-21T01:30:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 348666250, 2024-09-21T01:30:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 615206, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 8, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 4, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.389299879, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.075920350, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.246969830, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.050862756, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 1936, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.073503858, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000037966 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 2773, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 2725, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 307956271, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 1003975, 2024-09-21T01:30:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 6, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 5, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.437410933, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.129021771, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.250624212, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.056718450, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 2724, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.096730893, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000035510 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 1153920, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 304688 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 2855, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 3218, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 194921103, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 1716774, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 7, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 5, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.516356742, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.153028635, 2024-09-21T01:30:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.193771350, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.141999666, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 3216, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.135780113, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000042220 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 809520 2024-09-21T01:30:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "cct": { 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "total_workers": 1, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "unhealthy_workers": 0 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "finisher-MDSRank": { 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "queue_len": 0, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "complete_latency": { 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 11127, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.335216655, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000030126 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "finisher-PurgeQueue": { 2024-09-21T01:30:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "queue_len": 0, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "complete_latency": { 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "mds": { 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "request": 0, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "reply": 0, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "reply_latency": { 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "slow_reply": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "forward": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_fetch_complete": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_fetch_keys": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_commit": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_split": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_merge": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes": 32940, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_top": 11829, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_bottom": 21111, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_pin_tail": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_pinned": 16042, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_expired": 1, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_with_caps": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "caps": 0, 2024-09-21T01:30:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "subtrees": 1, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_hit": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_forward": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_discover": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_dir_fetch": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_remote_ino": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_lock": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "load_cent": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "q": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "exported": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "exported_inodes": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "imported": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "imported_inodes": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_dir_fetch": 0, 2024-09-21T01:30:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_backtrace_fetch": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_peer_discover": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rfiles": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rbytes": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rsnaps": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_set_tag": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_inotable_repaired": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dir_inodes": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_file_inodes": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_inode_file_caps": 0, 2024-09-21T01:30:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_grant": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_caps": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_caps_dirty": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_cap_release": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "process_request_cap_release": 0 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_cache": { 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_update": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_update_receipt": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_try_discover": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_send_discover": 0, 2024-09-21T01:30:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_handle_discover": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays_delayed": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays_enqueuing": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_created": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_enqueued": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_reintegrated": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_migrated": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_processing": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_enqueued": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_prioritized": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "recovery_started": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "recovery_completed": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_quiesce_path": 0, 2024-09-21T01:30:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_quiesce_inode": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_exportdir": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_flush": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_fragmentdir": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_fragstats": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_inodestats": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_started": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_succeeded": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_write_failed": 0 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_log": { 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "evlrg": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "evadd": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "evex": 0, 2024-09-21T01:30:15.743 INFO:teuthology.orchestra.run.smithi160.stdout: "evtrm": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "ev": 154046, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "evexg": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "evexd": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "segadd": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "segex": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "segtrm": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "seg": 135, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "segmjr": 14, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "segexg": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "segexd": 0, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "expos": 313110094, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "wrpos": 852538546, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "rdpos": 852523687, 2024-09-21T01:30:15.744 INFO:teuthology.orchestra.run.smithi160.stdout: "jlat": { 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "replayed": 154046 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_mem": { 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "ino": 3494, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "ino+": 32504, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "ino-": 29010, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "dir": 144, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "dir+": 197, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "dir-": 53, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "dn": 32940, 2024-09-21T01:30:15.745 INFO:teuthology.orchestra.run.smithi160.stdout: "dn+": 33264, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "dn-": 324, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "cap": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "cap+": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "cap-": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "rss": 162880, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "heap": 375304192 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_server": { 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "dispatch_client_request": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "dispatch_server_request": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_request": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_session": 0, 2024-09-21T01:30:15.746 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_peer_request": 0, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "req_create_latency": { 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getattr_latency": { 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getfilelock_latency": { 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "req_link_latency": { 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookup_latency": { 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookuphash_latency": { 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupino_latency": { 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupname_latency": { 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.749 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupparent_latency": { 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupsnap_latency": { 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lssnap_latency": { 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mkdir_latency": { 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.751 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mknod_latency": { 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mksnap_latency": { 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "req_open_latency": { 2024-09-21T01:30:15.752 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "req_readdir_latency": { 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rename_latency": { 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "req_renamesnap_latency": { 2024-09-21T01:30:15.753 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "req_snapdiff_latency": { 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmdir_latency": { 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.754 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmsnap_latency": { 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmxattr_latency": { 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setattr_latency": { 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setdirlayout_latency": { 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setfilelock_latency": { 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setlayout_latency": { 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.756 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setxattr_latency": { 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "req_symlink_latency": { 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "req_unlink_latency": { 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.757 INFO:teuthology.orchestra.run.smithi160.stdout: "cap_revoke_eviction": 0, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "cap_acquisition_throttle": 0, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getvxattr_latency": { 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_sessions": { 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "session_count": 2, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "session_add": 3, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "session_remove": 1, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "sessions_open": 2, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "sessions_stale": 0, 2024-09-21T01:30:15.758 INFO:teuthology.orchestra.run.smithi160.stdout: "total_load": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "average_load": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "avg_session_uptime": 3208, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "mdthresh_evicted": 0 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "mempool": { 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_bytes": 385965, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_items": 4000, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_bytes": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_items": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_bytes": 9976, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_items": 203, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_items": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_bytes": 31536861, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_items": 252497, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_items": 0, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_items": 0 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "objecter": { 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_active": 0, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_laggy": 0, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_send": 6985, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_send_bytes": 21, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_resend": 3, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_reply": 6985, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "op_latency": { 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 6985, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 22.360669979, 2024-09-21T01:30:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.003201241 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "op_inflight": 0, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "oplen_avg": { 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 6985, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 6986 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "op": 6985, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "op_r": 6985, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "op_w": 0, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "op_rmw": 0, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "op_pg": 0, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_stat": 4213, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_create": 0, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_read": 2770, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_write": 0, 2024-09-21T01:30:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_writefull": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_writesame": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_append": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_zero": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_truncate": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_delete": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_mapext": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_sparse_read": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_clonerange": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_getxattr": 1, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_setxattr": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_cmpxattr": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_rmxattr": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_resetxattrs": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_call": 0, 2024-09-21T01:30:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_watch": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_notify": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_pgls": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_pgls_filter": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_other": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_active": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_send": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_resend": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_ping": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_active": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_send": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_resend": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_active": 0, 2024-09-21T01:30:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_send": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_resend": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_active": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_send": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_resend": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "command_active": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "command_send": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "command_resend": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "map_epoch": 131, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "map_full": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "map_inc": 43, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_sessions": 11, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_session_open": 11, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_session_close": 0, 2024-09-21T01:30:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_laggy": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_wr": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_rd": 2, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_del": 0 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "oft": { 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_objs": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_kv_pairs": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_updates": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_removes": 0 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "purge_queue": { 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_ops": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing": 0, 2024-09-21T01:30:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_high_water": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executed_ops": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executed": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_item_in_journal": 0 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 7561, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 850830898, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 7561, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 7561, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 850830898, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-objecter_bytes": { 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 6985, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 848464756, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 6985, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 848464756, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-objecter_ops": { 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 1024, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 6985, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 6985, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 6985, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 6985, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-write_buf_throttle": { 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 3758096384, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-write_buf_throttle-0x561d5733b520": { 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 3758096384, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 0, 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 0, 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "trackedop": { 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "slow_ops_count": 0 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:15.775 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T01:30:16.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:16 smithi150 ceph-mon[34945]: pgmap v1232: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-09-21T01:30:16.359 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:16 smithi160 ceph-mon[34696]: pgmap v1232: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-09-21T01:30:16.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:16 smithi039 ceph-mon[30415]: pgmap v1232: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-09-21T01:30:16.534 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-21T01:30:16.534 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-21T01:30:16.534 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-09-21T01:30:16.534 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-09-21T01:30:16.534 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T01:30:16.534 DEBUG:tasks.check_counter:Getting stats from i 2024-09-21T01:30:16.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.i perf dump 2024-09-21T01:30:16.782 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:30:17.296 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T01:30:17.296 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:17.296 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 44546, 2024-09-21T01:30:17.296 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 44596, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 11265840, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 317664221, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 11, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 5, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 4.841091574, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 2.128249262, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.824493883, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 1.652791572, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 44594, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 3.194664356, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000071638 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:17.297 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 84, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000019409, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000231 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 3168, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 16272 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 321476, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 482380, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 101064128, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 770662477, 2024-09-21T01:30:17.298 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 13, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 6, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 25.915675727, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 15.781526700, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 11.714282932, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 1.730277005, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 482379, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 23.589981532, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000048903 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:17.299 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 345259, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.104189594, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000301 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 4032, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 721808 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 328659, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 318715, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 134212219, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 248297017, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 9, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 6, 2024-09-21T01:30:17.300 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 20.226476719, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 9.785172969, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 12.124059523, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.836687868, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 318714, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 15.492133359, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000048608 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 586121, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.093950999, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000160 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 1164144, 2024-09-21T01:30:17.301 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 308448 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "cct": { 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "total_workers": 1, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "unhealthy_workers": 0 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "finisher-MDSRank": { 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "queue_len": 0, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "complete_latency": { 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 79991, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 100.924635482, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.001261699 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.302 INFO:teuthology.orchestra.run.smithi160.stdout: "finisher-PurgeQueue": { 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "queue_len": 0, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "complete_latency": { 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 53739, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 7.363379068, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000137021 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "mds": { 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "request": 197957, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "reply": 191479, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "reply_latency": { 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 191479, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 1535.838661726, 2024-09-21T01:30:17.303 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.008020924 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "slow_reply": 0, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "forward": 6478, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_fetch_complete": 5, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_fetch_keys": 0, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_commit": 164, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_split": 10, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_merge": 4, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes": 4203, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_top": 1942, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_bottom": 2210, 2024-09-21T01:30:17.304 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_pin_tail": 51, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_pinned": 1427, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_expired": 674, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "inodes_with_caps": 2, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "caps": 2, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "subtrees": 4, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse": 292294, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_hit": 183451, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_forward": 6459, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_discover": 133, 2024-09-21T01:30:17.305 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_dir_fetch": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_remote_ino": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "traverse_lock": 2642, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "load_cent": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "q": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "exported": 15, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "exported_inodes": 312, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "imported": 36, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "imported_inodes": 1730, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_dir_fetch": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_backtrace_fetch": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "openino_peer_discover": 0, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rfiles": 702, 2024-09-21T01:30:17.306 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rbytes": 212015681, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "root_rsnaps": 6, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_set_tag": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_inotable_repaired": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dir_inodes": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "scrub_file_inodes": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_inode_file_caps": 32, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_revoke": 18701, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_grant": 128772, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T01:30:17.307 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_caps": 68326, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_caps_dirty": 26091, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_cap_release": 4897, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "process_request_cap_release": 145997 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_cache": { 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_update": 13, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_update_receipt": 82, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_try_discover": 48, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_send_discover": 211, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "dir_handle_discover": 184, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays": 41, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays_delayed": 0, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "num_strays_enqueuing": 0, 2024-09-21T01:30:17.308 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_created": 28800, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_enqueued": 28760, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_reintegrated": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "strays_migrated": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_processing": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_enqueued": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "num_recovering_prioritized": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "recovery_started": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "recovery_completed": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_quiesce_path": 25, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_quiesce_inode": 4080, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_exportdir": 19, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_flush": 0, 2024-09-21T01:30:17.309 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_fragmentdir": 14, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_fragstats": 0, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "ireq_inodestats": 0, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_started": 0, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_succeeded": 0, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "uninline_write_failed": 0 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_log": { 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "evlrg": 0, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "evadd": 154046, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "evex": 90367, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "evtrm": 86031, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "ev": 68015, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "evexg": 0, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "evexd": 4336, 2024-09-21T01:30:17.310 INFO:teuthology.orchestra.run.smithi160.stdout: "segadd": 253, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "segex": 125, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "segtrm": 118, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "seg": 135, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "segmjr": 6, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "segexg": 0, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "segexd": 7, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "expos": 313110094, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "wrpos": 852538546, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "rdpos": 0, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "jlat": { 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 44435, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 418.552294637, 2024-09-21T01:30:17.311 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.009419428 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "replayed": 0 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_mem": { 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "ino": 3454, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "ino+": 32619, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "ino-": 29165, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "dir": 170, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "dir+": 312, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "dir-": 142, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "dn": 4203, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "dn+": 56744, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "dn-": 52541, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "cap": 2, 2024-09-21T01:30:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "cap+": 30996, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "cap-": 30994, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "rss": 161240, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "heap": 371077120 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_server": { 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "dispatch_client_request": 301105, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "dispatch_server_request": 1553, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_request": 197957, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_client_session": 4193, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "handle_peer_request": 12236, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "req_create_latency": { 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 28407, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 322.463827864, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.011351562 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getattr_latency": { 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 1958, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 44.364182629, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.022657907 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getfilelock_latency": { 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "req_link_latency": { 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookup_latency": { 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 40990, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 108.411485766, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.002644827 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookuphash_latency": { 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupino_latency": { 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 1, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000219357, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000219357 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupname_latency": { 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupparent_latency": { 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lookupsnap_latency": { 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "req_lssnap_latency": { 2024-09-21T01:30:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mkdir_latency": { 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mknod_latency": { 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "req_mksnap_latency": { 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "req_open_latency": { 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 1, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.001056769, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.001056769 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "req_readdir_latency": { 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 83625, 2024-09-21T01:30:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 509.855713689, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.006096929 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rename_latency": { 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 6387, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 200.007472560, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.031314775 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "req_renamesnap_latency": { 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "req_snapdiff_latency": { 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.320 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmdir_latency": { 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 40, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.375095405, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.009377385 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmsnap_latency": { 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: "req_rmxattr_latency": { 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setattr_latency": { 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 202, 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 111.254299032, 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.550763856 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.322 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setdirlayout_latency": { 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setfilelock_latency": { 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 1108, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 1.167565181, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.001053759 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setlayout_latency": { 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "req_setxattr_latency": { 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "req_symlink_latency": { 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "req_unlink_latency": { 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 28760, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 237.937743474, 2024-09-21T01:30:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.008273217 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "cap_revoke_eviction": 0, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "cap_acquisition_throttle": 0, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "req_getvxattr_latency": { 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_sessions": { 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "session_count": 2, 2024-09-21T01:30:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: "session_add": 3, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "session_remove": 1, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "sessions_open": 2, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "sessions_stale": 0, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "total_load": 22694, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "average_load": 11347, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "avg_session_uptime": 1582, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "mdthresh_evicted": 0 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "mempool": { 2024-09-21T01:30:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_items": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_bytes": 705021, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_items": 1913, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_bytes": 88, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_items": 1, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_bytes": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_items": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_bytes": 9976, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_items": 203, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_items": 0, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_bytes": 11890740, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_items": 119781, 2024-09-21T01:30:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_items": 0, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_items": 0 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "objecter": { 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_active": 0, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_laggy": 0, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_send": 106418, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_send_bytes": 856982852, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_resend": 27, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_reply": 106418, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "op_latency": { 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 106418, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 1257.582462014, 2024-09-21T01:30:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.011817384 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "op_inflight": 0, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "oplen_avg": { 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 106418, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 108770 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "op": 106418, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "op_r": 1020, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "op_w": 105398, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "op_rmw": 0, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "op_pg": 0, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_stat": 34, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_create": 419, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_read": 1015, 2024-09-21T01:30:17.332 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_write": 45450, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_writefull": 605, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_writesame": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_append": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_zero": 1, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_truncate": 10, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_delete": 57819, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_mapext": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_sparse_read": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_clonerange": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_getxattr": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_setxattr": 654, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_cmpxattr": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_rmxattr": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_resetxattrs": 0, 2024-09-21T01:30:17.333 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_call": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_watch": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_notify": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_pgls": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_pgls_filter": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "osdop_other": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_active": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_send": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_resend": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "linger_ping": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_active": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_send": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "poolop_resend": 0, 2024-09-21T01:30:17.334 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_active": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_send": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "poolstat_resend": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_active": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_send": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "statfs_resend": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "command_active": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "command_send": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "command_resend": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "map_epoch": 131, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "map_full": 0, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "map_inc": 43, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_sessions": 12, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_session_open": 12, 2024-09-21T01:30:17.335 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_session_close": 0, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_laggy": 0, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_wr": 2035, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_rd": 10, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_del": 718 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "oft": { 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_objs": 1, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_kv_pairs": 0, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_updates": 0, 2024-09-21T01:30:17.336 INFO:teuthology.orchestra.run.smithi160.stdout: "omap_total_removes": 0 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "purge_queue": { 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_ops": 0, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_ops_high_water": 17, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing": 0, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executing_high_water": 9, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executed_ops": 57519, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_executed": 28760, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pq_item_in_journal": 0 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:17.337 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 694677, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 188461335, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 694677, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 694677, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 188461335, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-objecter_bytes": { 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 106418, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 860133844, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 106418, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 860133844, 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:17.339 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-objecter_ops": { 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 1024, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 0, 2024-09-21T01:30:17.340 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 106418, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 106418, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 106418, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 106418, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-write_buf_throttle": { 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 3758096384, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:17.341 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 28760, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 3151072, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 28760, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 1015, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 3151072, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.342 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-write_buf_throttle-0x5577a3ff9450": { 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 3758096384, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 154046, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 848344242, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 154046, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 44435, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 848344242, 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:17.343 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: "trackedop": { 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: "slow_ops_count": 2818 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:17.344 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T01:30:18.095 INFO:tasks.check_counter:Daemon mds.i mds.exported=15 2024-09-21T01:30:18.095 INFO:tasks.check_counter:Daemon mds.i mds.imported=36 2024-09-21T01:30:18.096 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-09-21T01:30:18.096 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-09-21T01:30:18.096 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T01:30:18.096 DEBUG:tasks.check_counter:Getting stats from l 2024-09-21T01:30:18.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph daemon mds.l perf dump 2024-09-21T01:30:18.344 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:30:18.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:18 smithi160 ceph-mon[34696]: pgmap v1233: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 10 op/s 2024-09-21T01:30:18.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:18 smithi039 ceph-mon[30415]: pgmap v1233: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 10 op/s 2024-09-21T01:30:18.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:18 smithi150 ceph-mon[34945]: pgmap v1233: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 10 op/s 2024-09-21T01:30:18.852 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T01:30:18.852 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T01:30:18.852 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 0, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 2, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 0, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 0, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 4, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 0, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.001150051, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.000000000, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.000311110, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.853 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 3, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 370, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 480, 2024-09-21T01:30:18.854 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 306400, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 5, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 2, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.026133722, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.021333386, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.001235724, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 368, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.024539345, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000066683 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:18.855 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 306400 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_messages": 491, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages": 484, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_bytes": 917232, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_bytes": 299296, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_created_connections": 2, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_active_connections": 1, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_total_time": 0.077977001, 2024-09-21T01:30:18.856 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_send_time": 0.039423538, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_recv_time": 0.047984754, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 484, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.028644320, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000059182 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_handle_ack_lat": { 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:18.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_recv_encrypted_bytes": 917232, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "msgr_send_encrypted_bytes": 299296 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "cct": { 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "total_workers": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "unhealthy_workers": 0 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "mempool": { 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_bytes": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bloom_filter_items": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_alloc_items": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_data_items": 0, 2024-09-21T01:30:18.858 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_other_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_extent_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_blob_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T01:30:18.859 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_fsck_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_txc_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluestore_writing_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_reader_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "bluefs_file_writer_items": 0, 2024-09-21T01:30:18.860 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_bytes": 148722, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_anon_items": 30, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "buffer_meta_items": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_items": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_mapbl_items": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osd_pglog_items": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_items": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "osdmap_mapping_items": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_bytes": 0, 2024-09-21T01:30:18.861 INFO:teuthology.orchestra.run.smithi160.stdout: "pgmap_items": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_bytes": 968, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "mds_co_items": 2, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_bytes": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_1_items": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_bytes": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "unittest_2_items": 0 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "max": 104857600, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "get_started": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "get": 490, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "get_sum": 845420, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_fail": 0, 2024-09-21T01:30:18.862 INFO:teuthology.orchestra.run.smithi160.stdout: "get_or_fail_success": 490, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "take": 0, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "take_sum": 0, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "put": 490, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "put_sum": 845420, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "wait": { 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "avgcount": 0, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "sum": 0.000000000, 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: "avgtime": 0.000000000 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T01:30:18.863 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T01:30:19.678 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-09-21T01:30:19.678 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-09-21T01:30:19.678 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-09-21T01:30:19.678 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-09-21T01:30:19.678 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-09-21T01:30:19.678 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-21T01:30:19.689 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-21T01:30:19.690 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:30:19.690 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:30:19.743 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T01:30:19.743 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:30:19.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:30:20.255 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:20 smithi160 ceph-mon[34696]: pgmap v1234: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-09-21T01:30:20.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:20 smithi160 ceph-mon[34696]: osdmap e132: 12 total, 12 up, 12 in 2024-09-21T01:30:20.394 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:20 smithi039 ceph-mon[30415]: pgmap v1234: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-09-21T01:30:20.394 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:20 smithi039 ceph-mon[30415]: osdmap e132: 12 total, 12 up, 12 in 2024-09-21T01:30:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:20 smithi150 ceph-mon[34945]: pgmap v1234: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-09-21T01:30:20.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:20 smithi150 ceph-mon[34945]: osdmap e132: 12 total, 12 up, 12 in 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 -- 172.21.15.39:0/3499383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4374103b60 msgr2=0x7f4374103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 --2- 172.21.15.39:0/3499383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4374103b60 0x7f4374103f70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4364009900 tx=0x7f436402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 -- 172.21.15.39:0/3499383592 shutdown_connections 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 --2- 172.21.15.39:0/3499383592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374109df0 0x7f4374110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 --2- 172.21.15.39:0/3499383592 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f43741094a0 0x7f43741098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 --2- 172.21.15.39:0/3499383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4374103b60 0x7f4374103f70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 -- 172.21.15.39:0/3499383592 >> 172.21.15.39:0/3499383592 conn(0x7f437407b850 msgr2=0x7f437407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 -- 172.21.15.39:0/3499383592 shutdown_connections 2024-09-21T01:30:20.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.887+0000 7f437a200640 1 -- 172.21.15.39:0/3499383592 wait complete. 2024-09-21T01:30:20.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.888+0000 7f437a200640 1 Processor -- start 2024-09-21T01:30:20.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 -- start start 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374103b60 0x7f437419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 0x7f437419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4374109df0 0x7f43741a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4374113b00 con 0x7f43741094a0 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4374113980 con 0x7f4374103b60 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f437a200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4374113c80 con 0x7f4374109df0 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f436be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 0x7f437419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f4373400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374103b60 0x7f437419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f436be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 0x7f437419cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39348/0 (socket says 172.21.15.39:39348) 2024-09-21T01:30:20.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f436be00640 1 -- 172.21.15.39:0/12803429 learned_addr learned my addr 172.21.15.39:0/12803429 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.889+0000 7f4373e00640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4374109df0 0x7f43741a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373400640 1 -- 172.21.15.39:0/12803429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4374109df0 msgr2=0x7f43741a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373400640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4374109df0 0x7f43741a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373400640 1 -- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 msgr2=0x7f437419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373400640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 0x7f437419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373400640 1 -- 172.21.15.39:0/12803429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43741a2640 con 0x7f4374103b60 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f436be00640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 0x7f437419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373e00640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4374109df0 0x7f43741a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.890+0000 7f4373400640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374103b60 0x7f437419c6d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f436402a9e0 tx=0x7f4364003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.891+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436403f070 con 0x7f4374103b60 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.891+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43741a28d0 con 0x7f4374103b60 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.891+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4364031b50 con 0x7f4374103b60 2024-09-21T01:30:20.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.891+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364035030 con 0x7f4374103b60 2024-09-21T01:30:20.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.891+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43741a9090 con 0x7f4374103b60 2024-09-21T01:30:20.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.892+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4374104de0 con 0x7f4374103b60 2024-09-21T01:30:20.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.895+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4364042070 con 0x7f4374103b60 2024-09-21T01:30:20.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.896+0000 7f4370c00640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f43580b6680 0x7f43580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:20.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.896+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9511+0+0 (secure 0 0 0) 0x7f43641086f0 con 0x7f4374103b60 2024-09-21T01:30:20.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.897+0000 7f436be00640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f43580b6680 0x7f43580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:20.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.897+0000 7f436be00640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f43580b6680 0x7f43580b8b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4374108f50 tx=0x7f435c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:20.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:20.898+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4364031d00 con 0x7f4374103b60 2024-09-21T01:30:21.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.109+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f43741a92f0 con 0x7f4374103b60 2024-09-21T01:30:21.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.110+0000 7f4370c00640 1 -- 172.21.15.39:0/12803429 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v132) ==== 81+0+884 (secure 0 0 0) 0x7f43640d2be0 con 0x7f4374103b60 2024-09-21T01:30:21.111 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2061408491 2024-09-22T00:53:07.774291+0000 2024-09-21T01:30:21.111 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2520758295 2024-09-22T00:51:28.880618+0000 2024-09-21T01:30:21.111 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2575866445 2024-09-22T00:51:58.924468+0000 2024-09-21T01:30:21.111 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/368668226 2024-09-22T00:51:28.880618+0000 2024-09-21T01:30:21.111 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3183273760 2024-09-22T00:53:07.774291+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3378583569 2024-09-22T00:51:58.924468+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/412418374 2024-09-22T00:51:28.880618+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2575866445 2024-09-22T00:51:58.924468+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/368668226 2024-09-22T00:51:28.880618+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3601289686 2024-09-22T00:51:58.924468+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/712381310 2024-09-22T00:51:58.924468+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3703447041 2024-09-22T00:51:28.880618+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/3953053829 2024-09-22T00:53:07.774291+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3894994060 2024-09-22T00:53:07.774291+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/3953053829 2024-09-22T00:53:07.774291+0000 2024-09-21T01:30:21.112 INFO:teuthology.orchestra.run.smithi039.stderr:listed 15 entries 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.112+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f43580b6680 msgr2=0x7f43580b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.112+0000 7f437a200640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f43580b6680 0x7f43580b8b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4374108f50 tx=0x7f435c005f90 comp rx=0 tx=0).stop 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374103b60 msgr2=0x7f437419c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374103b60 0x7f437419c6d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f436402a9e0 tx=0x7f4364003b90 comp rx=0 tx=0).stop 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 shutdown_connections 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4374109df0 0x7f43741a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43741094a0 0x7f437419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:21.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f43580b6680 0x7f43580b8b90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:21.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 --2- 172.21.15.39:0/12803429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4374103b60 0x7f437419c6d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:21.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 >> 172.21.15.39:0/12803429 conn(0x7f437407b850 msgr2=0x7f43741072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:21.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 shutdown_connections 2024-09-21T01:30:21.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:21.113+0000 7f437a200640 1 -- 172.21.15.39:0/12803429 wait complete. 2024-09-21T01:30:21.908 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-21T01:30:21.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:30:21.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-21T01:30:22.143 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:22 smithi039 ceph-mon[30415]: pgmap v1236: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-21T01:30:22.143 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:22 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/12803429' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:30:22.144 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:30:22.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:30:22.144 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-21T01:30:22.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:30:22.219 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-09-21T01:30:22.353 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-21T01:30:22.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:30:22.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:30:22.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:22 smithi160 ceph-mon[34696]: pgmap v1236: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-21T01:30:22.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:22 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/12803429' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:30:22.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:30:22.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-09-21T01:30:22.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:22 smithi150 ceph-mon[34945]: pgmap v1236: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-21T01:30:22.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:22 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/12803429' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:30:22.756 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T01:30:22.770 INFO:tasks.cephadm:Teardown begin 2024-09-21T01:30:22.770 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-21T01:30:22.770 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:30:23.176 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:24.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:24 smithi160 ceph-mon[34696]: pgmap v1237: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T01:30:24.398 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:24 smithi039 ceph-mon[30415]: pgmap v1237: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T01:30:24.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.489+0000 7f6569c00640 1 -- 172.21.15.39:0/3554161058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564105110 msgr2=0x7f656410d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:24.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.489+0000 7f6569c00640 1 --2- 172.21.15.39:0/3554161058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564105110 0x7f656410d660 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6550009900 tx=0x7f655002a410 comp rx=0 tx=0).stop 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 -- 172.21.15.39:0/3554161058 shutdown_connections 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 --2- 172.21.15.39:0/3554161058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656410dc20 0x7f6564110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 --2- 172.21.15.39:0/3554161058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564105110 0x7f656410d660 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 --2- 172.21.15.39:0/3554161058 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f65641047c0 0x7f6564104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 -- 172.21.15.39:0/3554161058 >> 172.21.15.39:0/3554161058 conn(0x7f65640fe2c0 msgr2=0x7f6564100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 -- 172.21.15.39:0/3554161058 shutdown_connections 2024-09-21T01:30:24.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.490+0000 7f6569c00640 1 -- 172.21.15.39:0/3554161058 wait complete. 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 Processor -- start 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 -- start start 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 0x7f656419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6564105110 0x7f656419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f656410dc20 0x7f656419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6564113840 con 0x7f65641047c0 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65641136c0 con 0x7f6564105110 2024-09-21T01:30:24.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.491+0000 7f6569c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f65641139c0 con 0x7f656410dc20 2024-09-21T01:30:24.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 0x7f656419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:24.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 0x7f656419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43108/0 (socket says 172.21.15.39:43108) 2024-09-21T01:30:24.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6564105110 0x7f656419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:24.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 -- 172.21.15.39:0/704660097 learned_addr learned my addr 172.21.15.39:0/704660097 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:24.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6569200640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f656410dc20 0x7f656419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 -- 172.21.15.39:0/704660097 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f656410dc20 msgr2=0x7f656419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f656410dc20 0x7f656419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 -- 172.21.15.39:0/704660097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6564105110 msgr2=0x7f656419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6564105110 0x7f656419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.492+0000 7f6563e00640 1 -- 172.21.15.39:0/704660097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65641a0280 con 0x7f65641047c0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.493+0000 7f6563e00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 0x7f656419a310 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f655400e9f0 tx=0x7f655400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.493+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655400cd80 con 0x7f65641047c0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.493+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65540108e0 con 0x7f65641047c0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.493+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65641a0570 con 0x7f65641047c0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.493+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65641a6cd0 con 0x7f65641047c0 2024-09-21T01:30:24.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.493+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554018980 con 0x7f65641047c0 2024-09-21T01:30:24.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.494+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65440052c0 con 0x7f65641047c0 2024-09-21T01:30:24.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.497+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6554010430 con 0x7f65641047c0 2024-09-21T01:30:24.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.500+0000 7f6560c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65480b6680 0x7f65480b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:24.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.500+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9511+0+0 (secure 0 0 0) 0x7f65540e7ed0 con 0x7f65641047c0 2024-09-21T01:30:24.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.500+0000 7f6563400640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65480b6680 0x7f65480b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:24.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.501+0000 7f6563400640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65480b6680 0x7f65480b8b90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6550009230 tx=0x7f6550031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.502+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f65540b23c0 con 0x7f65641047c0 2024-09-21T01:30:24.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:24 smithi150 ceph-mon[34945]: pgmap v1237: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T01:30:24.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.736+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65440055b0 con 0x7f65641047c0 2024-09-21T01:30:24.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.737+0000 7f6560c00640 1 -- 172.21.15.39:0/704660097 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+11915 (secure 0 0 0) 0x7f65540b7300 con 0x7f65641047c0 2024-09-21T01:30:24.739 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:30:24.739 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":68,"btime":"2024-09-21T01:29:39:840404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":68,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:29:39.419575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:30:24.739 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 68 2024-09-21T01:30:24.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65480b6680 msgr2=0x7f65480b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:24.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65480b6680 0x7f65480b8b90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6550009230 tx=0x7f6550031400 comp rx=0 tx=0).stop 2024-09-21T01:30:24.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 msgr2=0x7f656419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:24.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 0x7f656419a310 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f655400e9f0 tx=0x7f655400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 shutdown_connections 2024-09-21T01:30:24.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f656410dc20 0x7f656419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f65480b6680 0x7f65480b8b90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6564105110 0x7f656419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 --2- 172.21.15.39:0/704660097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65641047c0 0x7f656419a310 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 >> 172.21.15.39:0/704660097 conn(0x7f65640fe2c0 msgr2=0x7f65640feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 shutdown_connections 2024-09-21T01:30:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:24.740+0000 7f6569c00640 1 -- 172.21.15.39:0/704660097 wait complete. 2024-09-21T01:30:25.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:25 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/704660097' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:30:25.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:25 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/704660097' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:30:25.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:25 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/704660097' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:30:25.611 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:30:25.996 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:26.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:26 smithi160 ceph-mon[34696]: pgmap v1238: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-21T01:30:26.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:26 smithi039 ceph-mon[30415]: pgmap v1238: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-21T01:30:26.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:26 smithi150 ceph-mon[34945]: pgmap v1238: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 -- 172.21.15.39:0/661324372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810b6f0 msgr2=0x7fde2810bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 --2- 172.21.15.39:0/661324372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810b6f0 0x7fde2810bb00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fde10009900 tx=0x7fde1002a3f0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 -- 172.21.15.39:0/661324372 shutdown_connections 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 --2- 172.21.15.39:0/661324372 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde2810c040 0x7fde28112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 --2- 172.21.15.39:0/661324372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810b6f0 0x7fde2810bb00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 --2- 172.21.15.39:0/661324372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde28105db0 0x7fde281061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 -- 172.21.15.39:0/661324372 >> 172.21.15.39:0/661324372 conn(0x7fde281017e0 msgr2=0x7fde28103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 -- 172.21.15.39:0/661324372 shutdown_connections 2024-09-21T01:30:26.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.625+0000 7fde2ee00640 1 -- 172.21.15.39:0/661324372 wait complete. 2024-09-21T01:30:26.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.626+0000 7fde2ee00640 1 Processor -- start 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.626+0000 7fde2ee00640 1 -- start start 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ee00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde28105db0 0x7fde2819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ee00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2810b6f0 0x7fde2819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810c040 0x7fde281a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fde28115d00 con 0x7fde2810c040 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ee00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde28115b80 con 0x7fde2810b6f0 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ee00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fde28115e80 con 0x7fde28105db0 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde2ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810c040 0x7fde281a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde27e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde28105db0 0x7fde2819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde27e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde28105db0 0x7fde2819c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52236/0 (socket says 172.21.15.39:52236) 2024-09-21T01:30:26.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde1f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2810b6f0 0x7fde2819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:26.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.627+0000 7fde27e00640 1 -- 172.21.15.39:0/807680082 learned_addr learned my addr 172.21.15.39:0/807680082 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:26.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde1f400640 1 -- 172.21.15.39:0/807680082 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde28105db0 msgr2=0x7fde2819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde1f400640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde28105db0 0x7fde2819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde1f400640 1 -- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810c040 msgr2=0x7fde281a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde1f400640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810c040 0x7fde281a1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde1f400640 1 -- 172.21.15.39:0/807680082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde281a26e0 con 0x7fde2810b6f0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde2ce00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810c040 0x7fde281a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde1f400640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2810b6f0 0x7fde2819cd10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fde1002a910 tx=0x7fde10031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1003f070 con 0x7fde2810b6f0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.628+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde281a2970 con 0x7fde2810b6f0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.629+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde10004490 con 0x7fde2810b6f0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.629+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10035070 con 0x7fde2810b6f0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.629+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde281a9130 con 0x7fde2810b6f0 2024-09-21T01:30:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.630+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde281061c0 con 0x7fde2810b6f0 2024-09-21T01:30:26.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.632+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fde1003b460 con 0x7fde2810b6f0 2024-09-21T01:30:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.634+0000 7fde25600640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde0c0b6680 0x7fde0c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.634+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9511+0+0 (secure 0 0 0) 0x7fde10108200 con 0x7fde2810b6f0 2024-09-21T01:30:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.634+0000 7fde27e00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde0c0b6680 0x7fde0c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:26.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.635+0000 7fde27e00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde0c0b6680 0x7fde0c0b8b90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fde180095f0 tx=0x7fde18005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:26.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.636+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fde100d26f0 con 0x7fde2810b6f0 2024-09-21T01:30:26.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.879+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde2810de60 con 0x7fde2810b6f0 2024-09-21T01:30:26.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.880+0000 7fde25600640 1 -- 172.21.15.39:0/807680082 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+11915 (secure 0 0 0) 0x7fde100d7630 con 0x7fde2810b6f0 2024-09-21T01:30:26.881 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:30:26.882 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":68,"btime":"2024-09-21T01:29:39:840404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14667,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/319446739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":319446739},{"type":"v1","addr":"172.21.15.39:6835","nonce":319446739}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14733,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/443106775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":443106775},{"type":"v1","addr":"172.21.15.39:6839","nonce":443106775}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24671,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/845239112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":845239112},{"type":"v1","addr":"172.21.15.150:6833","nonce":845239112}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24673,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.160:6833/166386963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6832","nonce":166386963},{"type":"v1","addr":"172.21.15.160:6833","nonce":166386963}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2170415276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2170415276},{"type":"v1","addr":"172.21.15.150:6835","nonce":2170415276}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24703,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1544048762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1544048762},{"type":"v1","addr":"172.21.15.39:6837","nonce":1544048762}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":68,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T01:00:43.758917+0000","modified":"2024-09-21T01:29:39.419575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24764,"mds_1":24760,"mds_2":24742},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24715":{"gid":24715,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6835/2680876155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6834","nonce":2680876155},{"type":"v1","addr":"172.21.15.160:6835","nonce":2680876155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24725":{"gid":24725,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/555411539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":555411539},{"type":"v1","addr":"172.21.15.150:6837","nonce":555411539}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24727":{"gid":24727,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.160:6837/1965195431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6836","nonce":1965195431},{"type":"v1","addr":"172.21.15.160:6837","nonce":1965195431}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24742":{"gid":24742,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":21,"addr":"172.21.15.160:6839/2403371273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6838","nonce":2403371273},{"type":"v1","addr":"172.21.15.160:6839","nonce":2403371273}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24760":{"gid":24760,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":19,"addr":"172.21.15.39:6841/3868264331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3868264331},{"type":"v1","addr":"172.21.15.39:6841","nonce":3868264331}]},"join_fscid":-1,"export_targets":[0,2],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24764":{"gid":24764,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6839/103174700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":103174700},{"type":"v1","addr":"172.21.15.150:6839","nonce":103174700}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24764,"qdb_cluster":[24764,24760,24742]},"id":1}]} 2024-09-21T01:30:26.882 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 68 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde0c0b6680 msgr2=0x7fde0c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde0c0b6680 0x7fde0c0b8b90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fde180095f0 tx=0x7fde18005f90 comp rx=0 tx=0).stop 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2810b6f0 msgr2=0x7fde2819cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2810b6f0 0x7fde2819cd10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fde1002a910 tx=0x7fde10031a20 comp rx=0 tx=0).stop 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 shutdown_connections 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde2810c040 0x7fde281a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fde0c0b6680 0x7fde0c0b8b90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2810b6f0 0x7fde2819cd10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 --2- 172.21.15.39:0/807680082 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde28105db0 0x7fde2819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:26.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.882+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 >> 172.21.15.39:0/807680082 conn(0x7fde281017e0 msgr2=0x7fde2810ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:26.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.883+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 shutdown_connections 2024-09-21T01:30:26.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:26.883+0000 7fde2ee00640 1 -- 172.21.15.39:0/807680082 wait complete. 2024-09-21T01:30:27.153 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:27 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/807680082' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:30:27.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:27 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/807680082' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:30:27.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:27 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/807680082' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:30:27.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:30:27.994 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:28.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:28 smithi160 ceph-mon[34696]: pgmap v1239: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 8 op/s 2024-09-21T01:30:28.384 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:28 smithi039 ceph-mon[30415]: pgmap v1239: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 8 op/s 2024-09-21T01:30:28.465 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:28 smithi150 ceph-mon[34945]: pgmap v1239: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 8 op/s 2024-09-21T01:30:28.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 -- 172.21.15.39:0/648568855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14114750 msgr2=0x7f6b14116b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:28.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/648568855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14114750 0x7f6b14116b70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009ca0 tx=0x7f6b0c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 -- 172.21.15.39:0/648568855 shutdown_connections 2024-09-21T01:30:28.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/648568855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14114750 0x7f6b14116b70 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/648568855 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14078290 0x7f6b14114210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/648568855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b14077940 0x7f6b14077d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 -- 172.21.15.39:0/648568855 >> 172.21.15.39:0/648568855 conn(0x7f6b14101860 msgr2=0x7f6b14103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:28.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 -- 172.21.15.39:0/648568855 shutdown_connections 2024-09-21T01:30:28.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.709+0000 7f6b1aa00640 1 -- 172.21.15.39:0/648568855 wait complete. 2024-09-21T01:30:28.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.710+0000 7f6b1aa00640 1 Processor -- start 2024-09-21T01:30:28.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.710+0000 7f6b1aa00640 1 -- start start 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b1aa00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14077940 0x7f6b141a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b1aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 0x7f6b141a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b1aa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b14114750 0x7f6b141a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b1aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1411a0e0 con 0x7f6b14078290 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b1aa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6b14119f60 con 0x7f6b14114750 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b1aa00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1411a260 con 0x7f6b14077940 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b13400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 0x7f6b141a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b13e00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14077940 0x7f6b141a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b13400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 0x7f6b141a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43144/0 (socket says 172.21.15.39:43144) 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b13400640 1 -- 172.21.15.39:0/2821490629 learned_addr learned my addr 172.21.15.39:0/2821490629 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b18a00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b14114750 0x7f6b141a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.711+0000 7f6b13e00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14077940 0x7f6b141a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:52244/0 (socket says 172.21.15.39:52244) 2024-09-21T01:30:28.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b13400640 1 -- 172.21.15.39:0/2821490629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14077940 msgr2=0x7f6b141a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b13400640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14077940 0x7f6b141a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b13400640 1 -- 172.21.15.39:0/2821490629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b14114750 msgr2=0x7f6b141a62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b13400640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b14114750 0x7f6b141a62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b13400640 1 -- 172.21.15.39:0/2821490629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b141a69e0 con 0x7f6b14078290 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b13400640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 0x7f6b141a1010 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000e9c0 tx=0x7f6b0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0000ccc0 con 0x7f6b14078290 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b141a6cd0 con 0x7f6b14078290 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.712+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b000054d0 con 0x7f6b14078290 2024-09-21T01:30:28.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.713+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b141ad430 con 0x7f6b14078290 2024-09-21T01:30:28.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.713+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00021440 con 0x7f6b14078290 2024-09-21T01:30:28.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.714+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b14077d50 con 0x7f6b14078290 2024-09-21T01:30:28.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.716+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6b00004fa0 con 0x7f6b14078290 2024-09-21T01:30:28.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.718+0000 7f6b10c00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6af80b6680 0x7f6af80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:28.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.718+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9511+0+0 (secure 0 0 0) 0x7f6b000e82c0 con 0x7f6b14078290 2024-09-21T01:30:28.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.718+0000 7f6b13e00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6af80b6680 0x7f6af80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:28.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.719+0000 7f6b13e00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6af80b6680 0x7f6af80b8b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009bd0 tx=0x7f6b04005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.720+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6b000ae3d0 con 0x7f6b14078290 2024-09-21T01:30:28.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.923+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b141a1ba0 con 0x7f6b14078290 2024-09-21T01:30:28.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.924+0000 7f6b10c00640 1 -- 172.21.15.39:0/2821490629 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+25519 (secure 0 0 0) 0x7f6b141a1ba0 con 0x7f6b14078290 2024-09-21T01:30:28.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T01:30:28.926 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":132,"fsid":"61abfb08-77b3-11ef-baf5-efdc52797490","created":"2024-09-21T00:51:02.311192+0000","modified":"2024-09-21T01:30:19.773367+0000","last_up_change":"2024-09-21T00:59:43.981454+0000","last_in_change":"2024-09-21T00:59:21.240878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T00:55:22.265381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T01:00:37.542995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T01:00:40.673682+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":49,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T01:00:50.564539+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":49,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"db1d8552-f65b-468d-b4f6-a019f730b0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6803","nonce":1725836474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6805","nonce":1725836474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6809","nonce":1725836474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":1725836474},{"type":"v1","addr":"172.21.15.39:6807","nonce":1725836474}]},"public_addr":"172.21.15.39:6803/1725836474","cluster_addr":"172.21.15.39:6805/1725836474","heartbeat_back_addr":"172.21.15.39:6809/1725836474","heartbeat_front_addr":"172.21.15.39:6807/1725836474","state":["exists","up"]},{"osd":1,"uuid":"4a09b890-68cb-4443-b9eb-f1bf8557126e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6801","nonce":4107270363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6803","nonce":4107270363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6807","nonce":4107270363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4107270363},{"type":"v1","addr":"172.21.15.150:6805","nonce":4107270363}]},"public_addr":"172.21.15.150:6801/4107270363","cluster_addr":"172.21.15.150:6803/4107270363","heartbeat_back_addr":"172.21.15.150:6807/4107270363","heartbeat_front_addr":"172.21.15.150:6805/4107270363","state":["exists","up"]},{"osd":2,"uuid":"77400671-2855-4f44-911b-98ae802e9162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6801","nonce":4131806045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6803","nonce":4131806045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6807","nonce":4131806045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":4131806045},{"type":"v1","addr":"172.21.15.160:6805","nonce":4131806045}]},"public_addr":"172.21.15.160:6801/4131806045","cluster_addr":"172.21.15.160:6803/4131806045","heartbeat_back_addr":"172.21.15.160:6807/4131806045","heartbeat_front_addr":"172.21.15.160:6805/4131806045","state":["exists","up"]},{"osd":3,"uuid":"6f906a00-043e-4282-8c8d-47b889db0c65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6811","nonce":678333449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6813","nonce":678333449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6817","nonce":678333449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":678333449},{"type":"v1","addr":"172.21.15.39:6815","nonce":678333449}]},"public_addr":"172.21.15.39:6811/678333449","cluster_addr":"172.21.15.39:6813/678333449","heartbeat_back_addr":"172.21.15.39:6817/678333449","heartbeat_front_addr":"172.21.15.39:6815/678333449","state":["exists","up"]},{"osd":4,"uuid":"6a50d102-41d4-41ec-a5b8-1d40420d3cb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6809","nonce":1632873600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6811","nonce":1632873600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6815","nonce":1632873600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1632873600},{"type":"v1","addr":"172.21.15.150:6813","nonce":1632873600}]},"public_addr":"172.21.15.150:6809/1632873600","cluster_addr":"172.21.15.150:6811/1632873600","heartbeat_back_addr":"172.21.15.150:6815/1632873600","heartbeat_front_addr":"172.21.15.150:6813/1632873600","state":["exists","up"]},{"osd":5,"uuid":"64041b91-4bdc-4e60-b280-2407dc0285f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6809","nonce":3500100397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6811","nonce":3500100397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6815","nonce":3500100397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3500100397},{"type":"v1","addr":"172.21.15.160:6813","nonce":3500100397}]},"public_addr":"172.21.15.160:6809/3500100397","cluster_addr":"172.21.15.160:6811/3500100397","heartbeat_back_addr":"172.21.15.160:6815/3500100397","heartbeat_front_addr":"172.21.15.160:6813/3500100397","state":["exists","up"]},{"osd":6,"uuid":"2e0496c7-05ba-40e2-80fe-18ded790693a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6819","nonce":1468848111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6821","nonce":1468848111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6825","nonce":1468848111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1468848111},{"type":"v1","addr":"172.21.15.39:6823","nonce":1468848111}]},"public_addr":"172.21.15.39:6819/1468848111","cluster_addr":"172.21.15.39:6821/1468848111","heartbeat_back_addr":"172.21.15.39:6825/1468848111","heartbeat_front_addr":"172.21.15.39:6823/1468848111","state":["exists","up"]},{"osd":7,"uuid":"ec370951-5b96-470f-ad4d-1122615da035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6817","nonce":3982567281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6819","nonce":3982567281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6823","nonce":3982567281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3982567281},{"type":"v1","addr":"172.21.15.150:6821","nonce":3982567281}]},"public_addr":"172.21.15.150:6817/3982567281","cluster_addr":"172.21.15.150:6819/3982567281","heartbeat_back_addr":"172.21.15.150:6823/3982567281","heartbeat_front_addr":"172.21.15.150:6821/3982567281","state":["exists","up"]},{"osd":8,"uuid":"289d3eda-511f-4731-9242-02e987aafefd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6817","nonce":4075583447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6819","nonce":4075583447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6823","nonce":4075583447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":4075583447},{"type":"v1","addr":"172.21.15.160:6821","nonce":4075583447}]},"public_addr":"172.21.15.160:6817/4075583447","cluster_addr":"172.21.15.160:6819/4075583447","heartbeat_back_addr":"172.21.15.160:6823/4075583447","heartbeat_front_addr":"172.21.15.160:6821/4075583447","state":["exists","up"]},{"osd":9,"uuid":"eb6e99ce-4a27-484a-8850-52b165922de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6827","nonce":4176461767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6829","nonce":4176461767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6833","nonce":4176461767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":4176461767},{"type":"v1","addr":"172.21.15.39:6831","nonce":4176461767}]},"public_addr":"172.21.15.39:6827/4176461767","cluster_addr":"172.21.15.39:6829/4176461767","heartbeat_back_addr":"172.21.15.39:6833/4176461767","heartbeat_front_addr":"172.21.15.39:6831/4176461767","state":["exists","up"]},{"osd":10,"uuid":"cac8cd3a-cffc-466d-8214-8a3f714f5380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6825","nonce":2296139693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6827","nonce":2296139693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6831","nonce":2296139693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2296139693},{"type":"v1","addr":"172.21.15.150:6829","nonce":2296139693}]},"public_addr":"172.21.15.150:6825/2296139693","cluster_addr":"172.21.15.150:6827/2296139693","heartbeat_back_addr":"172.21.15.150:6831/2296139693","heartbeat_front_addr":"172.21.15.150:6829/2296139693","state":["exists","up"]},{"osd":11,"uuid":"f0574abe-bba5-4251-bfd6-2610607a2400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6825","nonce":3176075747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6827","nonce":3176075747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6831","nonce":3176075747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":3176075747},{"type":"v1","addr":"172.21.15.160:6829","nonce":3176075747}]},"public_addr":"172.21.15.160:6825/3176075747","cluster_addr":"172.21.15.160:6827/3176075747","heartbeat_back_addr":"172.21.15.160:6831/3176075747","heartbeat_front_addr":"172.21.15.160:6829/3176075747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:30.214966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:54:56.573026+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:20.843569+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:55:55.243795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:23.575796+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:56:49.736951+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:24.417080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:57:52.754303+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:18.611706+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:58:50.231238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:17.444285+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T00:59:42.205914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/2061408491":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/2520758295":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6801/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3183273760":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3378583569":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/412418374":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6801/2575866445":"2024-09-22T00:51:58.924468+0000","172.21.15.39:6800/368668226":"2024-09-22T00:51:28.880618+0000","172.21.15.39:0/3601289686":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/712381310":"2024-09-22T00:51:58.924468+0000","172.21.15.39:0/3703447041":"2024-09-22T00:51:28.880618+0000","172.21.15.39:6800/3953053829":"2024-09-22T00:53:07.774291+0000","172.21.15.39:0/3894994060":"2024-09-22T00:53:07.774291+0000","172.21.15.39:6801/3953053829":"2024-09-22T00:53:07.774291+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":36,"length":1},{"begin":49,"length":1}]},{"pool":4,"snaps":[{"begin":36,"length":1},{"begin":49,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:30:28.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6af80b6680 msgr2=0x7f6af80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:28.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6af80b6680 0x7f6af80b8b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009bd0 tx=0x7f6b04005f90 comp rx=0 tx=0).stop 2024-09-21T01:30:28.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 msgr2=0x7f6b141a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:28.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 0x7f6b141a1010 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000e9c0 tx=0x7f6b0000ee80 comp rx=0 tx=0).stop 2024-09-21T01:30:28.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 shutdown_connections 2024-09-21T01:30:28.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f6af80b6680 0x7f6af80b8b90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b14114750 0x7f6b141a62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b14078290 0x7f6b141a1010 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 --2- 172.21.15.39:0/2821490629 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b14077940 0x7f6b141a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:28.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 >> 172.21.15.39:0/2821490629 conn(0x7f6b14101860 msgr2=0x7f6b141167a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:28.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 shutdown_connections 2024-09-21T01:30:28.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:28.927+0000 7f6b1aa00640 1 -- 172.21.15.39:0/2821490629 wait complete. 2024-09-21T01:30:29.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:29 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2821490629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:30:29.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:29 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2821490629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:30:29.485 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:29 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2821490629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:30:29.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs fail cephfs 2024-09-21T01:30:30.077 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:30.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:30 smithi150 ceph-mon[34945]: pgmap v1240: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 8 op/s 2024-09-21T01:30:30.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:30 smithi160 ceph-mon[34696]: pgmap v1240: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 8 op/s 2024-09-21T01:30:30.394 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:30 smithi039 ceph-mon[30415]: pgmap v1240: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 8 op/s 2024-09-21T01:30:30.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.716+0000 7fb328c00640 1 -- 172.21.15.39:0/2232481104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb324103b80 msgr2=0x7fb324103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:30.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.716+0000 7fb328c00640 1 --2- 172.21.15.39:0/2232481104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb324103b80 0x7fb324103f90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb314009900 tx=0x7fb31402a3f0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 -- 172.21.15.39:0/2232481104 shutdown_connections 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 --2- 172.21.15.39:0/2232481104 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 0x7fb324110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 --2- 172.21.15.39:0/2232481104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3241094c0 0x7fb3241098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 --2- 172.21.15.39:0/2232481104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb324103b80 0x7fb324103f90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 -- 172.21.15.39:0/2232481104 >> 172.21.15.39:0/2232481104 conn(0x7fb3240ff5d0 msgr2=0x7fb324101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 -- 172.21.15.39:0/2232481104 shutdown_connections 2024-09-21T01:30:30.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.717+0000 7fb328c00640 1 -- 172.21.15.39:0/2232481104 wait complete. 2024-09-21T01:30:30.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.718+0000 7fb328c00640 1 Processor -- start 2024-09-21T01:30:30.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 -- start start 2024-09-21T01:30:30.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb324103b80 0x7fb32419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:30.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3241094c0 0x7fb32419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 0x7fb3241a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb324113b20 con 0x7fb3241094c0 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3241139a0 con 0x7fb324103b80 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb328c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb324113ca0 con 0x7fb324109e10 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb31be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3241094c0 0x7fb32419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.719+0000 7fb321e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb324103b80 0x7fb32419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 0x7fb3241a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb321e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb324103b80 0x7fb32419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39142/0 (socket says 172.21.15.39:39142) 2024-09-21T01:30:30.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb321e00640 1 -- 172.21.15.39:0/2896121254 learned_addr learned my addr 172.21.15.39:0/2896121254 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:30.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 -- 172.21.15.39:0/2896121254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb324103b80 msgr2=0x7fb32419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:30.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb324103b80 0x7fb32419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 -- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3241094c0 msgr2=0x7fb32419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:30.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3241094c0 0x7fb32419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 -- 172.21.15.39:0/2896121254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3241a2640 con 0x7fb324109e10 2024-09-21T01:30:30.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.720+0000 7fb322800640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 0x7fb3241a1f10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009fd0 tx=0x7fb31c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:30.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.721+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c00ce20 con 0x7fb324109e10 2024-09-21T01:30:30.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.721+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3241a2930 con 0x7fb324109e10 2024-09-21T01:30:30.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.721+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb31c005020 con 0x7fb324109e10 2024-09-21T01:30:30.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.721+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c010690 con 0x7fb324109e10 2024-09-21T01:30:30.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.721+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3241a9090 con 0x7fb324109e10 2024-09-21T01:30:30.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.722+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb324103f90 con 0x7fb324109e10 2024-09-21T01:30:30.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.725+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fb31c0054d0 con 0x7fb324109e10 2024-09-21T01:30:30.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.726+0000 7fb31aa00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb3080b6680 0x7fb3080b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:30.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.726+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9511+0+0 (secure 0 0 0) 0x7fb31c0e81f0 con 0x7fb324109e10 2024-09-21T01:30:30.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.726+0000 7fb321e00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb3080b6680 0x7fb3080b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:30.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.727+0000 7fb321e00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb3080b6680 0x7fb3080b8b90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb3140096c0 tx=0x7fb314005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:30.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.728+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb31c0ae300 con 0x7fb324109e10 2024-09-21T01:30:30.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.967+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb32419d800 con 0x7fb324109e10 2024-09-21T01:30:30.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.978+0000 7fb31aa00640 1 -- 172.21.15.39:0/2896121254 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) ==== 161+0+0 (secure 0 0 0) 0x7fb32419d800 con 0x7fb324109e10 2024-09-21T01:30:30.979 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb3080b6680 msgr2=0x7fb3080b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb3080b6680 0x7fb3080b8b90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb3140096c0 tx=0x7fb314005a20 comp rx=0 tx=0).stop 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 msgr2=0x7fb3241a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 0x7fb3241a1f10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009fd0 tx=0x7fb31c00ef10 comp rx=0 tx=0).stop 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 shutdown_connections 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb324109e10 0x7fb3241a1f10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3241094c0 0x7fb32419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7fb3080b6680 0x7fb3080b8b90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 --2- 172.21.15.39:0/2896121254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb324103b80 0x7fb32419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.980+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 >> 172.21.15.39:0/2896121254 conn(0x7fb3240ff5d0 msgr2=0x7fb324108b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:30.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.981+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 shutdown_connections 2024-09-21T01:30:30.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:30.981+0000 7fb328c00640 1 -- 172.21.15.39:0/2896121254 wait complete. 2024-09-21T01:30:31.120 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: ignoring --setuser ceph since I am not root 2024-09-21T01:30:31.120 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: ignoring --setgroup ceph since I am not root 2024-09-21T01:30:31.120 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: starting mds.b at 2024-09-21T01:30:31.120 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h[64165]: ignoring --setuser ceph since I am not root 2024-09-21T01:30:31.120 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h[64165]: ignoring --setgroup ceph since I am not root 2024-09-21T01:30:31.120 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mds[64169]: -- 172.21.15.150:0/225734123 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c3b6a1860 con 0x555c3f849600 2024-09-21T01:30:31.121 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h[64165]: starting mds.h at 2024-09-21T01:30:31.322 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a[30411]: 2024-09-21T01:30:30.968+0000 7f4861c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:30:31.322 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2896121254' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: osdmap e133: 12 total, 12 up, 12 in 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mon[30415]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a[72338]: ignoring --setuser ceph since I am not root 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a[72338]: ignoring --setgroup ceph since I am not root 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-mds[72344]: -- 172.21.15.39:0/3616969228 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55915c9ffc20 con 0x559160ba7180 2024-09-21T01:30:31.323 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:30:31 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a[72338]: starting mds.a at 2024-09-21T01:30:31.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2896121254' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T01:30:31.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: osdmap e133: 12 total, 12 up, 12 in 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mon[34696]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f[63967]: ignoring --setuser ceph since I am not root 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f[63967]: ignoring --setgroup ceph since I am not root 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mds[63971]: -- 172.21.15.160:0/1925861404 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8dc98d860 con 0x55f8e0b35180 2024-09-21T01:30:31.357 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f[63967]: starting mds.f at 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c[64241]: ignoring --setuser ceph since I am not root 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c[64241]: ignoring --setgroup ceph since I am not root 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mds[64245]: -- 172.21.15.160:0/1631484152 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601afe49c20 con 0x5601b3ff1a80 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c[64241]: starting mds.c at 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i[64525]: ignoring --setuser ceph since I am not root 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i[64525]: ignoring --setgroup ceph since I am not root 2024-09-21T01:30:31.358 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-mds[64529]: -- 172.21.15.160:0/378870485 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c65cdc3860 con 0x55c660f6b600 2024-09-21T01:30:31.359 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:30:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i[64525]: starting mds.i at 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2896121254' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: osdmap e133: 12 total, 12 up, 12 in 2024-09-21T01:30:31.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:31 smithi150 ceph-mon[34945]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-21T01:30:31.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-21T01:30:32.064 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: pgmap v1241: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 752 B/s wr, 7 op/s 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6834/3328651100,v1:172.21.15.160:6837/3328651100] up:boot 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6836/1824351502,v1:172.21.15.160:6839/1824351502] up:boot 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.160:6835/2813502069,v1:172.21.15.160:6838/2813502069] up:boot 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6836/204494517,v1:172.21.15.150:6838/204494517] up:boot 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.150:6837/374509828,v1:172.21.15.150:6839/374509828] up:boot 2024-09-21T01:30:32.379 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: mds.? [v2:172.21.15.39:6840/3394746564,v1:172.21.15.39:6841/3394746564] up:boot 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T01:30:32.380 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:32 smithi039 ceph-mon[30415]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T01:30:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: pgmap v1241: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 752 B/s wr, 7 op/s 2024-09-21T01:30:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6834/3328651100,v1:172.21.15.160:6837/3328651100] up:boot 2024-09-21T01:30:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6836/1824351502,v1:172.21.15.160:6839/1824351502] up:boot 2024-09-21T01:30:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.160:6835/2813502069,v1:172.21.15.160:6838/2813502069] up:boot 2024-09-21T01:30:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6836/204494517,v1:172.21.15.150:6838/204494517] up:boot 2024-09-21T01:30:32.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.150:6837/374509828,v1:172.21.15.150:6839/374509828] up:boot 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: mds.? [v2:172.21.15.39:6840/3394746564,v1:172.21.15.39:6841/3394746564] up:boot 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T01:30:32.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:32 smithi150 ceph-mon[34945]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: pgmap v1241: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 752 B/s wr, 7 op/s 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6834/3328651100,v1:172.21.15.160:6837/3328651100] up:boot 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6836/1824351502,v1:172.21.15.160:6839/1824351502] up:boot 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.160:6835/2813502069,v1:172.21.15.160:6838/2813502069] up:boot 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6836/204494517,v1:172.21.15.150:6838/204494517] up:boot 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.150:6837/374509828,v1:172.21.15.150:6839/374509828] up:boot 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: mds.? [v2:172.21.15.39:6840/3394746564,v1:172.21.15.39:6841/3394746564] up:boot 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T01:30:32.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T01:30:32.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T01:30:32.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T01:30:32.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T01:30:32.608 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:32 smithi160 ceph-mon[34696]: from='mgr.14223 172.21.15.39:0/577673310' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 -- 172.21.15.39:0/3572421336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 msgr2=0x7f470c110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 --2- 172.21.15.39:0/3572421336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 0x7f470c110450 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4704009ca0 tx=0x7f470402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 -- 172.21.15.39:0/3572421336 shutdown_connections 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 --2- 172.21.15.39:0/3572421336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 0x7f470c110450 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 --2- 172.21.15.39:0/3572421336 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f470c1092d0 0x7f470c1096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 --2- 172.21.15.39:0/3572421336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c103990 0x7f470c103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 -- 172.21.15.39:0/3572421336 >> 172.21.15.39:0/3572421336 conn(0x7f470c0ff3e0 msgr2=0x7f470c101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.708+0000 7f4710c00640 1 -- 172.21.15.39:0/3572421336 shutdown_connections 2024-09-21T01:30:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.709+0000 7f4710c00640 1 -- 172.21.15.39:0/3572421336 wait complete. 2024-09-21T01:30:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.710+0000 7f4710c00640 1 Processor -- start 2024-09-21T01:30:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.710+0000 7f4710c00640 1 -- start start 2024-09-21T01:30:32.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4710c00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f470c103990 0x7f470c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4710c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 0x7f470c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4710c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 0x7f470c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4710c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1138e0 con 0x7f470c109c20 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4710c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f470c113760 con 0x7f470c1092d0 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4710c00640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f470c113a60 con 0x7f470c103990 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4703e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 0x7f470c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f470be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 0x7f470c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4703e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 0x7f470c075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.39:39180/0 (socket says 172.21.15.39:39180) 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f4703e00640 1 -- 172.21.15.39:0/2728616821 learned_addr learned my addr 172.21.15.39:0/2728616821 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.711+0000 7f470b400640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f470c103990 0x7f470c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4703e00640 1 -- 172.21.15.39:0/2728616821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f470c103990 msgr2=0x7f470c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4703e00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f470c103990 0x7f470c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4703e00640 1 -- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 msgr2=0x7f470c076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4703e00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 0x7f470c076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4703e00640 1 -- 172.21.15.39:0/2728616821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c1a8570 con 0x7f470c1092d0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4703e00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 0x7f470c075c50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46f400e9c0 tx=0x7f46f400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f400ccc0 con 0x7f470c1092d0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c1a8860 con 0x7f470c1092d0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46f40054d0 con 0x7f470c1092d0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4010430 con 0x7f470c1092d0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.712+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c1a8d50 con 0x7f470c1092d0 2024-09-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.713+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c103da0 con 0x7f470c1092d0 2024-09-21T01:30:32.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.726+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f46f4005000 con 0x7f470c1092d0 2024-09-21T01:30:32.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.727+0000 7f4708c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46f00b6680 0x7f46f00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:32.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.727+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9681+0+0 (secure 0 0 0) 0x7f46f40e8110 con 0x7f470c1092d0 2024-09-21T01:30:32.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.727+0000 7f470b400640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46f00b6680 0x7f46f00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:32.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.728+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46f4056710 con 0x7f470c1092d0 2024-09-21T01:30:32.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.728+0000 7f470b400640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46f00b6680 0x7f46f00b8b90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f470c077160 tx=0x7f46fc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:32.965+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f470c02e640 con 0x7f470c1092d0 2024-09-21T01:30:33.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.135+0000 7f4708c00640 1 -- 172.21.15.39:0/2728616821 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v71) ==== 104+0+0 (secure 0 0 0) 0x7f46f40052f0 con 0x7f470c1092d0 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46f00b6680 msgr2=0x7f46f00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46f00b6680 0x7f46f00b8b90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f470c077160 tx=0x7f46fc005f90 comp rx=0 tx=0).stop 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 msgr2=0x7f470c075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 0x7f470c075c50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46f400e9c0 tx=0x7f46f400ee80 comp rx=0 tx=0).stop 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 shutdown_connections 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f470c109c20 0x7f470c076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:33.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f46f00b6680 0x7f46f00b8b90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:33.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c1092d0 0x7f470c075c50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:33.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 --2- 172.21.15.39:0/2728616821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f470c103990 0x7f470c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:33.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 >> 172.21.15.39:0/2728616821 conn(0x7f470c0ff3e0 msgr2=0x7f470c101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:33.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 shutdown_connections 2024-09-21T01:30:33.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:33.138+0000 7f4710c00640 1 -- 172.21.15.39:0/2728616821 wait complete. 2024-09-21T01:30:33.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:33 smithi039 ceph-mon[30415]: pgmap v1243: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 7 op/s 2024-09-21T01:30:33.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:33 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2728616821' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:30:33.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:33 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:30:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:33 smithi150 ceph-mon[34945]: pgmap v1243: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 7 op/s 2024-09-21T01:30:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:33 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2728616821' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:30:33.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:33 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:30:33.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:33 smithi160 ceph-mon[34696]: pgmap v1243: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 7 op/s 2024-09-21T01:30:33.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:33 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2728616821' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:30:33.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:33 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:30:33.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-21T01:30:34.238 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:34.376 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:34 smithi039 ceph-mon[30415]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T01:30:34.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:34 smithi039 ceph-mon[30415]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:30:34.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:34 smithi039 ceph-mon[30415]: Cluster is now healthy 2024-09-21T01:30:34.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:34 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:30:34.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:34 smithi039 ceph-mon[30415]: fsmap 12 up:standby 2024-09-21T01:30:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:34 smithi150 ceph-mon[34945]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T01:30:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:34 smithi150 ceph-mon[34945]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:30:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:34 smithi150 ceph-mon[34945]: Cluster is now healthy 2024-09-21T01:30:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:34 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:30:34.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:34 smithi150 ceph-mon[34945]: fsmap 12 up:standby 2024-09-21T01:30:34.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:34 smithi160 ceph-mon[34696]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T01:30:34.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:34 smithi160 ceph-mon[34696]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:30:34.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:34 smithi160 ceph-mon[34696]: Cluster is now healthy 2024-09-21T01:30:34.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:34 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:30:34.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:34 smithi160 ceph-mon[34696]: fsmap 12 up:standby 2024-09-21T01:30:34.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 -- 172.21.15.39:0/375135975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d98105bb0 msgr2=0x7f3d98105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:34.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 --2- 172.21.15.39:0/375135975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d98105bb0 0x7f3d98105fc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009900 tx=0x7f3d8802a3f0 comp rx=0 tx=0).stop 2024-09-21T01:30:34.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 -- 172.21.15.39:0/375135975 shutdown_connections 2024-09-21T01:30:34.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 --2- 172.21.15.39:0/375135975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9810be40 0x7f3d98112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:34.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 --2- 172.21.15.39:0/375135975 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 0x7f3d9810b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:34.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 --2- 172.21.15.39:0/375135975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d98105bb0 0x7f3d98105fc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:34.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 -- 172.21.15.39:0/375135975 >> 172.21.15.39:0/375135975 conn(0x7f3d98101680 msgr2=0x7f3d98103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:34.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 -- 172.21.15.39:0/375135975 shutdown_connections 2024-09-21T01:30:34.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.920+0000 7f3d9d000640 1 -- 172.21.15.39:0/375135975 wait complete. 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.921+0000 7f3d9d000640 1 Processor -- start 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.921+0000 7f3d9d000640 1 -- start start 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d9d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d98105bb0 0x7f3d9819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d9d000640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 0x7f3d9819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d9d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 0x7f3d981a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d9d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3d98115b50 con 0x7f3d9810be40 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d9d000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981159d0 con 0x7f3d98105bb0 2024-09-21T01:30:34.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d9d000640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3d98115cd0 con 0x7f3d9810b4f0 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 0x7f3d981a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 0x7f3d981a1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37346/0 (socket says 172.21.15.39:37346) 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 -- 172.21.15.39:0/3079392224 learned_addr learned my addr 172.21.15.39:0/3079392224 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d8ea00640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 0x7f3d9819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97400640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d98105bb0 0x7f3d9819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 -- 172.21.15.39:0/3079392224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 msgr2=0x7f3d9819cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 0x7f3d9819cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:34.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 -- 172.21.15.39:0/3079392224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d98105bb0 msgr2=0x7f3d9819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.922+0000 7f3d97e00640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d98105bb0 0x7f3d9819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d97e00640 1 -- 172.21.15.39:0/3079392224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d981a24b0 con 0x7f3d9810be40 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d8ea00640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 0x7f3d9819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d97e00640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 0x7f3d981a1d80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000eb10 tx=0x7f3d9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90004b20 con 0x7f3d9810be40 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d90004cc0 con 0x7f3d9810be40 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90010670 con 0x7f3d9810be40 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d981a27a0 con 0x7f3d9810be40 2024-09-21T01:30:34.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.923+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d981a8f00 con 0x7f3d9810be40 2024-09-21T01:30:34.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.924+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d98105fc0 con 0x7f3d9810be40 2024-09-21T01:30:34.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.927+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3d90005000 con 0x7f3d9810be40 2024-09-21T01:30:34.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.929+0000 7f3d94c00640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3d7c0b6700 0x7f3d7c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:34.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.929+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9681+0+0 (secure 0 0 0) 0x7f3d900e82d0 con 0x7f3d9810be40 2024-09-21T01:30:34.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.929+0000 7f3d97400640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3d7c0b6700 0x7f3d7c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:34.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.930+0000 7f3d97400640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3d7c0b6700 0x7f3d7c0b8c10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d88005ab0 tx=0x7f3d88005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:34.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:34.931+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3d90056710 con 0x7f3d9810be40 2024-09-21T01:30:35.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.128+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3d9819d670 con 0x7f3d9810be40 2024-09-21T01:30:35.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.141+0000 7f3d94c00640 1 -- 172.21.15.39:0/3079392224 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) ==== 181+0+0 (secure 0 0 0) 0x7f3d9819d670 con 0x7f3d9810be40 2024-09-21T01:30:35.142 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3d7c0b6700 msgr2=0x7f3d7c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3d7c0b6700 0x7f3d7c0b8c10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d88005ab0 tx=0x7f3d88005a20 comp rx=0 tx=0).stop 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 msgr2=0x7f3d981a1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 0x7f3d981a1d80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000eb10 tx=0x7f3d9000c2d0 comp rx=0 tx=0).stop 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 shutdown_connections 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d9810be40 0x7f3d981a1d80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:35.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3d9810b4f0 0x7f3d9819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:35.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f3d7c0b6700 0x7f3d7c0b8c10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:35.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 --2- 172.21.15.39:0/3079392224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d98105bb0 0x7f3d9819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:35.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 >> 172.21.15.39:0/3079392224 conn(0x7f3d98101680 msgr2=0x7f3d98103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:35.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 shutdown_connections 2024-09-21T01:30:35.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:35.144+0000 7f3d9d000640 1 -- 172.21.15.39:0/3079392224 wait complete. 2024-09-21T01:30:35.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:35 smithi039 ceph-mon[30415]: pgmap v1244: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 409 B/s wr, 5 op/s 2024-09-21T01:30:35.404 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:35 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3079392224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:35 smithi150 ceph-mon[34945]: pgmap v1244: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 409 B/s wr, 5 op/s 2024-09-21T01:30:35.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:35 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3079392224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:35.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:35 smithi160 ceph-mon[34696]: pgmap v1244: 129 pgs: 129 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 409 B/s wr, 5 op/s 2024-09-21T01:30:35.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:35 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3079392224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:35.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-21T01:30:36.308 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:36.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:36 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3079392224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:36.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:36 smithi039 ceph-mon[30415]: osdmap e134: 12 total, 12 up, 12 in 2024-09-21T01:30:36.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:36 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3079392224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:36.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:36 smithi150 ceph-mon[34945]: osdmap e134: 12 total, 12 up, 12 in 2024-09-21T01:30:36.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:36 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3079392224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:36.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:36 smithi160 ceph-mon[34696]: osdmap e134: 12 total, 12 up, 12 in 2024-09-21T01:30:36.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 -- 172.21.15.39:0/2038676319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 msgr2=0x7f6474112670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:36.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 --2- 172.21.15.39:0/2038676319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 0x7f6474112670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f646c009ca0 tx=0x7f646c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:30:36.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 -- 172.21.15.39:0/2038676319 shutdown_connections 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 --2- 172.21.15.39:0/2038676319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 0x7f6474112670 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 --2- 172.21.15.39:0/2038676319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647410b4f0 0x7f647410b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 --2- 172.21.15.39:0/2038676319 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6474105bb0 0x7f6474105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 -- 172.21.15.39:0/2038676319 >> 172.21.15.39:0/2038676319 conn(0x7f6474101680 msgr2=0x7f6474103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 -- 172.21.15.39:0/2038676319 shutdown_connections 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.978+0000 7f6473200640 1 -- 172.21.15.39:0/2038676319 wait complete. 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.979+0000 7f6473200640 1 Processor -- start 2024-09-21T01:30:36.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 -- start start 2024-09-21T01:30:36.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474105bb0 0x7f647419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:36.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 0x7f647419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:36.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 0x7f64741a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:36.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6474115b50 con 0x7f647410be40 2024-09-21T01:30:36.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64741159d0 con 0x7f6474105bb0 2024-09-21T01:30:36.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6473200640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6474115cd0 con 0x7f647410b4f0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f646be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 0x7f647419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f646be00640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 0x7f647419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.39:56056/0 (socket says 172.21.15.39:56056) 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f646be00640 1 -- 172.21.15.39:0/2772197024 learned_addr learned my addr 172.21.15.39:0/2772197024 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6472800640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 0x7f64741a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.980+0000 7f6471e00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474105bb0 0x7f647419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646be00640 1 -- 172.21.15.39:0/2772197024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474105bb0 msgr2=0x7f647419c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646be00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474105bb0 0x7f647419c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646be00640 1 -- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 msgr2=0x7f64741a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646be00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 0x7f64741a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646be00640 1 -- 172.21.15.39:0/2772197024 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64741a2450 con 0x7f647410b4f0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646be00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 0x7f647419ca80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f645c00e9f0 tx=0x7f645c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c00ccc0 con 0x7f647410b4f0 2024-09-21T01:30:36.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64741a2740 con 0x7f647410b4f0 2024-09-21T01:30:36.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f645c0108e0 con 0x7f647410b4f0 2024-09-21T01:30:36.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.981+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c018860 con 0x7f647410b4f0 2024-09-21T01:30:36.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.982+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64741a8ea0 con 0x7f647410b4f0 2024-09-21T01:30:36.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.983+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6474105fc0 con 0x7f647410b4f0 2024-09-21T01:30:36.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.985+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f645c010430 con 0x7f647410b4f0 2024-09-21T01:30:36.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.987+0000 7f646aa00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64580b6680 0x7f64580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:36.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.987+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9185+0+0 (secure 0 0 0) 0x7f645c0e80f0 con 0x7f647410b4f0 2024-09-21T01:30:36.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.987+0000 7f6471e00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64580b6680 0x7f64580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:36.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.988+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f645c0b2720 con 0x7f647410b4f0 2024-09-21T01:30:36.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:36.988+0000 7f6471e00640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64580b6680 0x7f64580b8b90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6464009bd0 tx=0x7f6464005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:37.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:37.189+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f647419d610 con 0x7f647410b4f0 2024-09-21T01:30:37.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:37 smithi039 ceph-mon[30415]: pgmap v1246: 97 pgs: 97 active+clean; 300 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:37.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:37 smithi150 ceph-mon[34945]: pgmap v1246: 97 pgs: 97 active+clean; 300 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:37.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:37 smithi160 ceph-mon[34696]: pgmap v1246: 97 pgs: 97 active+clean; 300 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-09-21T01:30:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.170+0000 7f646aa00640 1 -- 172.21.15.39:0/2772197024 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) ==== 169+0+0 (secure 0 0 0) 0x7f647419d610 con 0x7f647410b4f0 2024-09-21T01:30:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.175+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64580b6680 msgr2=0x7f64580b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.175+0000 7f6473200640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64580b6680 0x7f64580b8b90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6464009bd0 tx=0x7f6464005f90 comp rx=0 tx=0).stop 2024-09-21T01:30:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.175+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 msgr2=0x7f647419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.175+0000 7f6473200640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 0x7f647419ca80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f645c00e9f0 tx=0x7f645c00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 shutdown_connections 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f647410be40 0x7f64741a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f647410b4f0 0x7f647419ca80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7f64580b6680 0x7f64580b8b90 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 --2- 172.21.15.39:0/2772197024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474105bb0 0x7f647419c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 >> 172.21.15.39:0/2772197024 conn(0x7f6474101680 msgr2=0x7f6474108b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.176+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 shutdown_connections 2024-09-21T01:30:38.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:38.178+0000 7f6473200640 1 -- 172.21.15.39:0/2772197024 wait complete. 2024-09-21T01:30:38.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:38 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/2772197024' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:38.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:38 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:38 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/2772197024' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:38.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:38 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:38.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:38 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/2772197024' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:38.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:38 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:38.903 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-21T01:30:39.302 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:30:39.440 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:39 smithi039 ceph-mon[30415]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:39.440 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:39 smithi039 ceph-mon[30415]: osdmap e135: 12 total, 12 up, 12 in 2024-09-21T01:30:39.440 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:39 smithi039 ceph-mon[30415]: pgmap v1248: 33 pgs: 33 active+clean; 300 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:39 smithi150 ceph-mon[34945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:39 smithi150 ceph-mon[34945]: osdmap e135: 12 total, 12 up, 12 in 2024-09-21T01:30:39.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:39 smithi150 ceph-mon[34945]: pgmap v1248: 33 pgs: 33 active+clean; 300 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:39.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:39 smithi160 ceph-mon[34696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:39 smithi160 ceph-mon[34696]: osdmap e135: 12 total, 12 up, 12 in 2024-09-21T01:30:39.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:39 smithi160 ceph-mon[34696]: pgmap v1248: 33 pgs: 33 active+clean; 300 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 -- 172.21.15.39:0/248694902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810c010 msgr2=0x7ff328112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 --2- 172.21.15.39:0/248694902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810c010 0x7ff328112840 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff320009ca0 tx=0x7ff32002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 -- 172.21.15.39:0/248694902 shutdown_connections 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 --2- 172.21.15.39:0/248694902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810c010 0x7ff328112840 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 --2- 172.21.15.39:0/248694902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810b6c0 0x7ff32810bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 --2- 172.21.15.39:0/248694902 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff328105d80 0x7ff328106190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:39.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.915+0000 7ff330400640 1 -- 172.21.15.39:0/248694902 >> 172.21.15.39:0/248694902 conn(0x7ff328101810 msgr2=0x7ff328103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:39.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.916+0000 7ff330400640 1 -- 172.21.15.39:0/248694902 shutdown_connections 2024-09-21T01:30:39.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.916+0000 7ff330400640 1 -- 172.21.15.39:0/248694902 wait complete. 2024-09-21T01:30:39.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.917+0000 7ff330400640 1 Processor -- start 2024-09-21T01:30:39.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.917+0000 7ff330400640 1 -- start start 2024-09-21T01:30:39.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff330400640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff328105d80 0x7ff32819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:39.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff330400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 0x7ff32819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:39.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff330400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810c010 0x7ff3281a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:39.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff330400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff328115cd0 con 0x7ff32810b6c0 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff330400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff328115b50 con 0x7ff32810c010 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff330400640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff328115e50 con 0x7ff328105d80 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff32d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 0x7ff32819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff32d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 0x7ff32819cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37384/0 (socket says 172.21.15.39:37384) 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff32d000640 1 -- 172.21.15.39:0/3717515444 learned_addr learned my addr 172.21.15.39:0/3717515444 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff32da00640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff328105d80 0x7ff32819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:39.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.918+0000 7ff32e400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810c010 0x7ff3281a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32d000640 1 -- 172.21.15.39:0/3717515444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff328105d80 msgr2=0x7ff32819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32d000640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff328105d80 0x7ff32819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32d000640 1 -- 172.21.15.39:0/3717515444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810c010 msgr2=0x7ff3281a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32d000640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810c010 0x7ff3281a1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32d000640 1 -- 172.21.15.39:0/3717515444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3281a25d0 con 0x7ff32810b6c0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32e400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810c010 0x7ff3281a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff32d000640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 0x7ff32819cc00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff31400e9f0 tx=0x7ff31400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31400ccc0 con 0x7ff32810b6c0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3140108e0 con 0x7ff32810b6c0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff314018860 con 0x7ff32810b6c0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3281a28c0 con 0x7ff32810b6c0 2024-09-21T01:30:39.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.919+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3281a9020 con 0x7ff32810b6c0 2024-09-21T01:30:39.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.920+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff328106190 con 0x7ff32810b6c0 2024-09-21T01:30:39.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.924+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff314010430 con 0x7ff32810b6c0 2024-09-21T01:30:39.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.925+0000 7ff326000640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3100b6700 0x7ff3100b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:30:39.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.925+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 8539+0+0 (secure 0 0 0) 0x7ff3140e7ef0 con 0x7ff32810b6c0 2024-09-21T01:30:39.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.925+0000 7ff32da00640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3100b6700 0x7ff3100b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:30:39.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.926+0000 7ff32da00640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3100b6700 0x7ff3100b8c10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009bd0 tx=0x7ff31c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:30:39.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:39.926+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff3140ae3d0 con 0x7ff32810b6c0 2024-09-21T01:30:40.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.123+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff32819d790 con 0x7ff32810b6c0 2024-09-21T01:30:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data_ec' removed 2024-09-21T01:30:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.186+0000 7ff326000640 1 -- 172.21.15.39:0/3717515444 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v136) ==== 178+0+0 (secure 0 0 0) 0x7ff32819d790 con 0x7ff32810b6c0 2024-09-21T01:30:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3100b6700 msgr2=0x7ff3100b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3100b6700 0x7ff3100b8c10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009bd0 tx=0x7ff31c005f90 comp rx=0 tx=0).stop 2024-09-21T01:30:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 msgr2=0x7ff32819cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:30:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 0x7ff32819cc00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff31400e9f0 tx=0x7ff31400eeb0 comp rx=0 tx=0).stop 2024-09-21T01:30:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 shutdown_connections 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:6800/2923702906,v1:172.21.15.39:6801/2923702906] conn(0x7ff3100b6700 0x7ff3100b8c10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32810c010 0x7ff3281a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff32810b6c0 0x7ff32819cc00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 --2- 172.21.15.39:0/3717515444 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff328105d80 0x7ff32819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 >> 172.21.15.39:0/3717515444 conn(0x7ff328101810 msgr2=0x7ff328108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 shutdown_connections 2024-09-21T01:30:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T01:30:40.189+0000 7ff330400640 1 -- 172.21.15.39:0/3717515444 wait complete. 2024-09-21T01:30:40.356 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:40 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3717515444' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:40.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:40 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3717515444' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:40.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:40 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3717515444' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T01:30:41.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:30:41.329 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:30:41.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:30:41.412 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T01:30:41.412 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:30:41.448 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:30:41.464 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:30:41.479 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T01:30:41.479 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T01:30:41.479 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a 2024-09-21T01:30:41.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:41 smithi039 ceph-mon[30415]: from='client.? 172.21.15.39:0/3717515444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:41.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:41 smithi039 ceph-mon[30415]: osdmap e136: 12 total, 12 up, 12 in 2024-09-21T01:30:41.535 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:41 smithi039 ceph-mon[30415]: pgmap v1250: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:41.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:41 smithi150 ceph-mon[34945]: from='client.? 172.21.15.39:0/3717515444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:41.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:41 smithi150 ceph-mon[34945]: osdmap e136: 12 total, 12 up, 12 in 2024-09-21T01:30:41.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:41 smithi150 ceph-mon[34945]: pgmap v1250: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:41.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:41 smithi160 ceph-mon[34696]: from='client.? 172.21.15.39:0/3717515444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T01:30:41.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:41 smithi160 ceph-mon[34696]: osdmap e136: 12 total, 12 up, 12 in 2024-09-21T01:30:41.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:41 smithi160 ceph-mon[34696]: pgmap v1250: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T01:30:41.847 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:41 smithi039 systemd[1]: Stopping Ceph mon.a for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:42.147 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a[30411]: 2024-09-21T01:30:42.012+0000 7f4868a00640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:30:42.147 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a[30411]: 2024-09-21T01:30:42.013+0000 7f4868a00640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T01:30:42.403 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:42 smithi039 podman[107568]: 2024-09-21 01:30:42.147052398 +0000 UTC m=+0.314669706 container died 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T01:30:42.845 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:42 smithi039 podman[107568]: 2024-09-21 01:30:42.618046574 +0000 UTC m=+0.785663880 container cleanup 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T01:30:42.845 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:42 smithi039 bash[107568]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a 2024-09-21T01:30:43.107 INFO:journalctl@ceph.mon.a.smithi039.stdout:Sep 21 01:30:42 smithi039 podman[107581]: 2024-09-21 01:30:42.844468213 +0000 UTC m=+0.690787028 container remove 74510f689f9d4e4ae66dd843b32f4999ff7b85403571370ffd6dd08282405cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T01:30:43.116 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.a.service' 2024-09-21T01:30:43.220 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:43.220 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T01:30:43.220 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-21T01:30:43.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.b 2024-09-21T01:30:43.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:43 smithi150 systemd[1]: Stopping Ceph mon.b for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:44.058 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b[34941]: 2024-09-21T01:30:43.598+0000 7f17b121d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:30:44.058 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b[34941]: 2024-09-21T01:30:43.598+0000 7f17b121d640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T01:30:44.059 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:43 smithi150 podman[72758]: 2024-09-21 01:30:43.740983779 +0000 UTC m=+0.265766443 container died 39ea9691389c269a0e1d9058e9f05c03ce1a3930198999b7cb7596d635085032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:30:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:44 smithi150 podman[72758]: 2024-09-21 01:30:44.165963117 +0000 UTC m=+0.690745781 container cleanup 39ea9691389c269a0e1d9058e9f05c03ce1a3930198999b7cb7596d635085032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T01:30:44.314 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:44 smithi150 bash[72758]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b 2024-09-21T01:30:44.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.b.service' 2024-09-21T01:30:44.650 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:44 smithi150 podman[72770]: 2024-09-21 01:30:44.384026888 +0000 UTC m=+0.644041732 container remove 39ea9691389c269a0e1d9058e9f05c03ce1a3930198999b7cb7596d635085032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:30:44.650 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:44 smithi150 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T01:30:44.650 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:44 smithi150 systemd[1]: Stopped Ceph mon.b for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:30:44.650 INFO:journalctl@ceph.mon.b.smithi150.stdout:Sep 21 01:30:44 smithi150 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.b.service: Consumed 33.296s CPU time. 2024-09-21T01:30:44.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:44.679 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-21T01:30:44.679 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-21T01:30:44.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.c 2024-09-21T01:30:45.106 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:44 smithi160 systemd[1]: Stopping Ceph mon.c for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-c[34692]: 2024-09-21T01:30:45.121+0000 7fd1e4521640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:30:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-c[34692]: 2024-09-21T01:30:45.121+0000 7fd1e4521640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T01:30:45.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:45 smithi160 podman[73956]: 2024-09-21 01:30:45.265958679 +0000 UTC m=+0.316768546 container died 5c44cd059d0ef5807e44c897e34cf3657f01e378241e3dea438a689af02f2afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:30:46.339 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:46 smithi160 podman[73956]: 2024-09-21 01:30:46.104225737 +0000 UTC m=+1.155035603 container cleanup 5c44cd059d0ef5807e44c897e34cf3657f01e378241e3dea438a689af02f2afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T01:30:46.339 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:46 smithi160 bash[73956]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-c 2024-09-21T01:30:46.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.c.service' 2024-09-21T01:30:46.593 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:46 smithi160 podman[73968]: 2024-09-21 01:30:46.339170167 +0000 UTC m=+1.069317225 container remove 5c44cd059d0ef5807e44c897e34cf3657f01e378241e3dea438a689af02f2afd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mon-c, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:30:46.594 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:46 smithi160 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T01:30:46.594 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:46 smithi160 systemd[1]: Stopped Ceph mon.c for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:30:46.594 INFO:journalctl@ceph.mon.c.smithi160.stdout:Sep 21 01:30:46 smithi160 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mon.c.service: Consumed 23.851s CPU time. 2024-09-21T01:30:46.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:46.615 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-21T01:30:46.615 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T01:30:46.616 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x 2024-09-21T01:30:47.044 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:30:46 smithi039 systemd[1]: Stopping Ceph mgr.x for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:47.403 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:30:47 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x[30637]: 2024-09-21T01:30:47.041+0000 7f4e4a600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:30:47.403 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:30:47 smithi039 podman[107685]: 2024-09-21 01:30:47.145462259 +0000 UTC m=+0.240403910 container died 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:30:47.841 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:30:47 smithi039 podman[107685]: 2024-09-21 01:30:47.622508613 +0000 UTC m=+0.717450262 container cleanup 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True) 2024-09-21T01:30:47.841 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:30:47 smithi039 bash[107685]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x 2024-09-21T01:30:48.105 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Sep 21 01:30:47 smithi039 podman[107698]: 2024-09-21 01:30:47.840714586 +0000 UTC m=+0.688008237 container remove 9dd56f0e66318e40c88173f27e07c490bcaed0e0f5d35257501772424b4b6316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:30:48.114 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-21T01:30:48.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:48.189 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T01:30:48.189 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T01:30:48.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.y 2024-09-21T01:30:48.534 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 01:30:48 smithi150 systemd[1]: Stopping Ceph mgr.y for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:48.814 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 01:30:48 smithi150 podman[72881]: 2024-09-21 01:30:48.534044131 +0000 UTC m=+0.147701640 container died 9814ec731fc18385c65c6540cd622309a41deb552d4b89a4b274c5482bf64ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:30:49.173 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 01:30:48 smithi150 podman[72881]: 2024-09-21 01:30:48.963676777 +0000 UTC m=+0.577334284 container cleanup 9814ec731fc18385c65c6540cd622309a41deb552d4b89a4b274c5482bf64ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-y, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T01:30:49.173 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Sep 21 01:30:48 smithi150 bash[72881]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-y 2024-09-21T01:30:49.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-21T01:30:49.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:49.354 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T01:30:49.354 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-21T01:30:49.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.z 2024-09-21T01:30:49.696 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 01:30:49 smithi160 systemd[1]: Stopping Ceph mgr.z for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:49.968 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 01:30:49 smithi160 podman[74074]: 2024-09-21 01:30:49.69610805 +0000 UTC m=+0.139166388 container died 143afbe3a5566adf734851f2bcf669d5407d588c2dc0d70a354e0d0e1958c9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-z, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:30:50.356 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 01:30:50 smithi160 podman[74074]: 2024-09-21 01:30:50.067255592 +0000 UTC m=+0.510313921 container cleanup 143afbe3a5566adf734851f2bcf669d5407d588c2dc0d70a354e0d0e1958c9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-z, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T01:30:50.357 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Sep 21 01:30:50 smithi160 bash[74074]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mgr-z 2024-09-21T01:30:50.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mgr.z.service' 2024-09-21T01:30:50.580 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:50.580 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-21T01:30:50.580 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T01:30:50.580 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.0 2024-09-21T01:30:50.903 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:50 smithi039 systemd[1]: Stopping Ceph osd.0 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:30:51.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0[46832]: 2024-09-21T01:30:51.021+0000 7f533f600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:30:51.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0[46832]: 2024-09-21T01:30:51.021+0000 7f533f600640 -1 osd.0 136 *** Got signal Terminated *** 2024-09-21T01:30:51.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0[46832]: 2024-09-21T01:30:51.021+0000 7f533f600640 -1 osd.0 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:30:56.903 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:56 smithi039 podman[107806]: 2024-09-21 01:30:56.602798498 +0000 UTC m=+5.730383673 container died 3927d12efc3778833c852283db5b2635adb62fec7f631a3f201948bc38704015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T01:30:57.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:57 smithi039 podman[107806]: 2024-09-21 01:30:57.027437271 +0000 UTC m=+6.155022450 container cleanup 3927d12efc3778833c852283db5b2635adb62fec7f631a3f201948bc38704015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:30:57.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:57 smithi039 bash[107806]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0 2024-09-21T01:30:57.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:57 smithi039 podman[107823]: 2024-09-21 01:30:57.236876639 +0000 UTC m=+0.628553322 container remove 3927d12efc3778833c852283db5b2635adb62fec7f631a3f201948bc38704015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T01:30:58.082 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:57 smithi039 podman[107911]: 2024-09-21 01:30:57.699574433 +0000 UTC m=+0.027708406 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:30:58.082 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:57 smithi039 podman[107911]: 2024-09-21 01:30:57.821140999 +0000 UTC m=+0.149274949 container create edb80055b8fdcd426667be3ece12eef235e6d91cd1a2820c01f4d8ba1bb6ea7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_BRANCH=HEAD) 2024-09-21T01:30:58.082 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:58 smithi039 podman[107911]: 2024-09-21 01:30:58.082218143 +0000 UTC m=+0.410352123 container init edb80055b8fdcd426667be3ece12eef235e6d91cd1a2820c01f4d8ba1bb6ea7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T01:30:58.358 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:58 smithi039 podman[107911]: 2024-09-21 01:30:58.089291986 +0000 UTC m=+0.417425938 container start edb80055b8fdcd426667be3ece12eef235e6d91cd1a2820c01f4d8ba1bb6ea7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T01:30:58.358 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:58 smithi039 podman[107911]: 2024-09-21 01:30:58.190263327 +0000 UTC m=+0.518397277 container attach edb80055b8fdcd426667be3ece12eef235e6d91cd1a2820c01f4d8ba1bb6ea7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T01:30:58.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Sep 21 01:30:58 smithi039 podman[107911]: 2024-09-21 01:30:58.358905102 +0000 UTC m=+0.687039062 container died edb80055b8fdcd426667be3ece12eef235e6d91cd1a2820c01f4d8ba1bb6ea7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T01:30:59.115 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.0.service' 2024-09-21T01:30:59.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:30:59.194 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T01:30:59.194 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T01:30:59.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.1 2024-09-21T01:30:59.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:30:59 smithi150 systemd[1]: Stopping Ceph osd.1 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:00.064 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:30:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1[39575]: 2024-09-21T01:30:59.644+0000 7f61f772d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:00.064 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:30:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1[39575]: 2024-09-21T01:30:59.644+0000 7f61f772d640 -1 osd.1 136 *** Got signal Terminated *** 2024-09-21T01:31:00.064 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:30:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1[39575]: 2024-09-21T01:30:59.644+0000 7f61f772d640 -1 osd.1 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:05.530 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:05 smithi150 podman[73004]: 2024-09-21 01:31:05.219780594 +0000 UTC m=+5.830171597 container died 69d8995f2833cef9a0a42da85f86b901649d7c5c156ef20165fc543d56b2d67a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T01:31:05.814 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:05 smithi150 podman[73004]: 2024-09-21 01:31:05.669584785 +0000 UTC m=+6.279975788 container cleanup 69d8995f2833cef9a0a42da85f86b901649d7c5c156ef20165fc543d56b2d67a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T01:31:05.814 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:05 smithi150 bash[73004]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1 2024-09-21T01:31:06.304 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:05 smithi150 podman[73032]: 2024-09-21 01:31:05.996373404 +0000 UTC m=+0.772114145 container remove 69d8995f2833cef9a0a42da85f86b901649d7c5c156ef20165fc543d56b2d67a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T01:31:06.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:06 smithi150 podman[73119]: 2024-09-21 01:31:06.207878851 +0000 UTC m=+0.016209472 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:06.564 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:06 smithi150 podman[73119]: 2024-09-21 01:31:06.32230516 +0000 UTC m=+0.130635818 container create 91ccfd4fe74b4debdd35eb4197e4eaddc3149c5eb779fec7ef1d6528d93367c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0) 2024-09-21T01:31:06.909 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:06 smithi150 podman[73119]: 2024-09-21 01:31:06.579641992 +0000 UTC m=+0.387972606 container init 91ccfd4fe74b4debdd35eb4197e4eaddc3149c5eb779fec7ef1d6528d93367c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:31:06.909 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:06 smithi150 podman[73119]: 2024-09-21 01:31:06.583082746 +0000 UTC m=+0.391413359 container start 91ccfd4fe74b4debdd35eb4197e4eaddc3149c5eb779fec7ef1d6528d93367c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:31:06.909 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:06 smithi150 podman[73119]: 2024-09-21 01:31:06.700300204 +0000 UTC m=+0.508630823 container attach 91ccfd4fe74b4debdd35eb4197e4eaddc3149c5eb779fec7ef1d6528d93367c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T01:31:07.270 INFO:journalctl@ceph.osd.1.smithi150.stdout:Sep 21 01:31:06 smithi150 podman[73119]: 2024-09-21 01:31:06.909433353 +0000 UTC m=+0.717763967 container died 91ccfd4fe74b4debdd35eb4197e4eaddc3149c5eb779fec7ef1d6528d93367c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T01:31:07.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.1.service' 2024-09-21T01:31:07.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:07.619 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T01:31:07.619 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T01:31:07.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.2 2024-09-21T01:31:08.032 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:07 smithi160 systemd[1]: Stopping Ceph osd.2 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:08.356 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2[39902]: 2024-09-21T01:31:08.058+0000 7fb9ab923640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:08.357 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2[39902]: 2024-09-21T01:31:08.058+0000 7fb9ab923640 -1 osd.2 136 *** Got signal Terminated *** 2024-09-21T01:31:08.357 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2[39902]: 2024-09-21T01:31:08.058+0000 7fb9ab923640 -1 osd.2 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:13.815 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:13 smithi160 podman[74201]: 2024-09-21 01:31:13.525571497 +0000 UTC m=+5.712551921 container died ec22a90618a2e777dceda906452072d4ab6479c110caae09199ac76995f0a014 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T01:31:14.107 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:13 smithi160 podman[74201]: 2024-09-21 01:31:13.913284514 +0000 UTC m=+6.100264937 container cleanup ec22a90618a2e777dceda906452072d4ab6479c110caae09199ac76995f0a014 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T01:31:14.107 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:13 smithi160 bash[74201]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2 2024-09-21T01:31:14.466 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:14 smithi160 podman[74227]: 2024-09-21 01:31:14.1569591 +0000 UTC m=+0.624828864 container remove ec22a90618a2e777dceda906452072d4ab6479c110caae09199ac76995f0a014 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:31:14.761 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:14 smithi160 podman[74317]: 2024-09-21 01:31:14.36867284 +0000 UTC m=+0.015937598 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:14.761 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:14 smithi160 podman[74317]: 2024-09-21 01:31:14.499743769 +0000 UTC m=+0.147008526 container create 87412aec28d7f218ca4b0faf176c725b8bcd28b8b73354d891d8a1a3ca742083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:31:14.761 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:14 smithi160 podman[74317]: 2024-09-21 01:31:14.760716284 +0000 UTC m=+0.407981033 container init 87412aec28d7f218ca4b0faf176c725b8bcd28b8b73354d891d8a1a3ca742083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-21T01:31:15.022 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:14 smithi160 podman[74317]: 2024-09-21 01:31:14.764572268 +0000 UTC m=+0.411837028 container start 87412aec28d7f218ca4b0faf176c725b8bcd28b8b73354d891d8a1a3ca742083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T01:31:15.022 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:14 smithi160 podman[74317]: 2024-09-21 01:31:14.860837342 +0000 UTC m=+0.508102107 container attach 87412aec28d7f218ca4b0faf176c725b8bcd28b8b73354d891d8a1a3ca742083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2) 2024-09-21T01:31:15.022 INFO:journalctl@ceph.osd.2.smithi160.stdout:Sep 21 01:31:15 smithi160 podman[74317]: 2024-09-21 01:31:15.00008802 +0000 UTC m=+0.647352784 container died 87412aec28d7f218ca4b0faf176c725b8bcd28b8b73354d891d8a1a3ca742083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T01:31:15.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.2.service' 2024-09-21T01:31:15.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:15.718 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T01:31:15.718 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T01:31:15.718 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.3 2024-09-21T01:31:16.154 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:15 smithi039 systemd[1]: Stopping Ceph osd.3 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:16.653 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:16 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3[53265]: 2024-09-21T01:31:16.272+0000 7fadbd000640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:16.653 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:16 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3[53265]: 2024-09-21T01:31:16.272+0000 7fadbd000640 -1 osd.3 136 *** Got signal Terminated *** 2024-09-21T01:31:16.653 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:16 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3[53265]: 2024-09-21T01:31:16.272+0000 7fadbd000640 -1 osd.3 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:17.764+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:18.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:18 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:18.416+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:18.724+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:19.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:18.783+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:19.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:19 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:19.382+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:19.820 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:19.449+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:20.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:19 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:19.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:20.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:19.819+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:20.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:20 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:20.345+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:20.799 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:20.490+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:21.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:20 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:20.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:21.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:20.799+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:21.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:21 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:21.302+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:21.838 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:21.445+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:22.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:21 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:21.730+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:22.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:21.838+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:22.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:22.054+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:22.307 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:22 smithi039 podman[108036]: 2024-09-21 01:31:22.001545546 +0000 UTC m=+5.980011258 container died 998a0e3dbd36999ecfb42588e953530ce27f7f605025fdea5eaa10cccd7ea31e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T01:31:22.565 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:22 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:22.323+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:22.565 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:22 smithi039 podman[108036]: 2024-09-21 01:31:22.547328965 +0000 UTC m=+6.525794692 container cleanup 998a0e3dbd36999ecfb42588e953530ce27f7f605025fdea5eaa10cccd7ea31e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T01:31:22.565 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:22 smithi039 bash[108036]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3 2024-09-21T01:31:22.816 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:22.492+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:23.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:22 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:22.714+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:23.106 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:23.044+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:23.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:22.816+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:23.153 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:22 smithi039 podman[108054]: 2024-09-21 01:31:22.765475828 +0000 UTC m=+0.751936716 container remove 998a0e3dbd36999ecfb42588e953530ce27f7f605025fdea5eaa10cccd7ea31e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T01:31:23.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:23 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:23.320+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:23.654 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:23 smithi039 podman[108141]: 2024-09-21 01:31:23.223937451 +0000 UTC m=+0.028113148 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:23.654 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:23 smithi039 podman[108141]: 2024-09-21 01:31:23.360616592 +0000 UTC m=+0.164792277 container create 640fd8345f9ed8ab275756cb7fc177d8441fd9512e06a70b2af4568027fb6ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T01:31:23.781 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:23.495+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:23.939 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:23 smithi039 podman[108141]: 2024-09-21 01:31:23.660636881 +0000 UTC m=+0.464812566 container init 640fd8345f9ed8ab275756cb7fc177d8441fd9512e06a70b2af4568027fb6ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T01:31:23.939 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:23 smithi039 podman[108141]: 2024-09-21 01:31:23.667441277 +0000 UTC m=+0.471616962 container start 640fd8345f9ed8ab275756cb7fc177d8441fd9512e06a70b2af4568027fb6ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T01:31:23.939 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:23 smithi039 podman[108141]: 2024-09-21 01:31:23.735382685 +0000 UTC m=+0.539558383 container attach 640fd8345f9ed8ab275756cb7fc177d8441fd9512e06a70b2af4568027fb6ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T01:31:23.939 INFO:journalctl@ceph.osd.3.smithi039.stdout:Sep 21 01:31:23 smithi039 podman[108141]: 2024-09-21 01:31:23.939003183 +0000 UTC m=+0.743178883 container died 640fd8345f9ed8ab275756cb7fc177d8441fd9512e06a70b2af4568027fb6ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:31:24.063 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:23 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:23.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:24.069 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:23.781+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:24.356 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:24.069+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:24.564 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:24 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:24.367+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:24.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.3.service' 2024-09-21T01:31:24.670 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:24.670 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T01:31:24.671 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T01:31:24.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.4 2024-09-21T01:31:24.831 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:24.450+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:24.999 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:24 smithi150 systemd[1]: Stopping Ceph osd.4 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:24.999 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:24 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:24.690+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:25.102 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:24.831+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:25.314 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:24 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:24.998+0000 7f0fe0704640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:25.314 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:24 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:24.998+0000 7f0fe0704640 -1 osd.4 136 *** Got signal Terminated *** 2024-09-21T01:31:25.314 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:24 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:24.998+0000 7f0fe0704640 -1 osd.4 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:25.356 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:25.104+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:25.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:25 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:25.364+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:25.802 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:25.407+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:26.010 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:25 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:25.728+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:26.080 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:25.802+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:26.313 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:26 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:26.009+0000 7f0fd8ce7640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:30:59.434510+0000 front 2024-09-21T01:30:59.434486+0000 (oldest deadline 2024-09-21T01:31:25.334704+0000) 2024-09-21T01:31:26.356 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:26.079+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:26.356 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:26.079+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:26.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:26 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:26.362+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:26.765 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:26.371+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:26.765 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:26.371+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:27.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:26 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:26.979+0000 7f0fd8ce7640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:30:59.434510+0000 front 2024-09-21T01:30:59.434486+0000 (oldest deadline 2024-09-21T01:31:25.334704+0000) 2024-09-21T01:31:27.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:26 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:26.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:27.097 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:26.765+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:27.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:26 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:26.858+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:27.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:27.097+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:27.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:27.097+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:27.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:27 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:27.354+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:27.799 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:27.383+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:27.799 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:27.383+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:28.014 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:27 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:27.759+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:28.014 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:27 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:27.759+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:28.014 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:27 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:27.763+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:28.059 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:27.799+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:28.059 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:27.799+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:28.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:27 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:27.848+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:28.314 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:28 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:28.013+0000 7f0fd8ce7640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:30:59.434510+0000 front 2024-09-21T01:30:59.434486+0000 (oldest deadline 2024-09-21T01:31:25.334704+0000) 2024-09-21T01:31:28.356 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:28 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:28.058+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:28.356 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:28 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:28.058+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:28.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:28 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:28.387+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:28.808 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:28 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:28.385+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:28.808 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:28 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:28.385+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:29.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:28 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4[46138]: 2024-09-21T01:31:28.968+0000 7f0fd8ce7640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:30:59.434510+0000 front 2024-09-21T01:30:59.434486+0000 (oldest deadline 2024-09-21T01:31:25.334704+0000) 2024-09-21T01:31:29.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:28 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:28.730+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:29.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:28 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:28.727+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:29.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:28 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:28.727+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:29.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:28 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:28.807+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:29.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:28 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:28.807+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:29.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:29.015+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:29.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:29.015+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:29.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:28 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:28.885+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:29.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:29 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:29.349+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:29.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:29 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:29.349+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:29.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:29.377+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:29.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:29.377+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:30.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:29 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:29.691+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:30.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:29 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:29.721+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:30.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:29 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:29.721+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:30.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:29 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:29.863+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:30.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:29.997+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:30.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:29.997+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:30.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:29.857+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:30.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:29 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:29.857+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:30.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:30.363+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:30.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:30.363+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:30.677 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:30 smithi150 podman[73222]: 2024-09-21 01:31:30.419816331 +0000 UTC m=+5.551481504 container died 66df1b80b32c947b15179392613c339799b3d9ebc2d2982f22e6723d641c6f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:31:30.835 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:30 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:30.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:30.835 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:30 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:30.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:31.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:30 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:30.682+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:31.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:30 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:30.682+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:31.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:30 smithi150 podman[73222]: 2024-09-21 01:31:30.791033135 +0000 UTC m=+5.922698300 container cleanup 66df1b80b32c947b15179392613c339799b3d9ebc2d2982f22e6723d641c6f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T01:31:31.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:30 smithi150 bash[73222]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4 2024-09-21T01:31:31.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:30 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:30.677+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:31.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:30 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:30.834+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:31.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:30 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:30.834+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:31.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:31.034+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:31.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:31.034+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:31.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:30.842+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:31.398 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73237]: 2024-09-21 01:31:31.084016772 +0000 UTC m=+0.658638324 container remove 66df1b80b32c947b15179392613c339799b3d9ebc2d2982f22e6723d641c6f7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T01:31:31.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:31 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:31.328+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:31.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:31 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:31.328+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:31.662 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73327]: 2024-09-21 01:31:31.300999293 +0000 UTC m=+0.015859643 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:31.662 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73327]: 2024-09-21 01:31:31.409799133 +0000 UTC m=+0.124659477 container create ebcec9f7a9c22c1295f2c16f388563a4998fbe9a60c60afe9b9cac67819d1770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0) 2024-09-21T01:31:31.663 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:31.629+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:31.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:31.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:31.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:31.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:32.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73327]: 2024-09-21 01:31:31.662673963 +0000 UTC m=+0.377534316 container init ebcec9f7a9c22c1295f2c16f388563a4998fbe9a60c60afe9b9cac67819d1770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T01:31:32.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73327]: 2024-09-21 01:31:31.666347703 +0000 UTC m=+0.381208048 container start ebcec9f7a9c22c1295f2c16f388563a4998fbe9a60c60afe9b9cac67819d1770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:31:32.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73327]: 2024-09-21 01:31:31.779158124 +0000 UTC m=+0.494018487 container attach ebcec9f7a9c22c1295f2c16f388563a4998fbe9a60c60afe9b9cac67819d1770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T01:31:32.064 INFO:journalctl@ceph.osd.4.smithi150.stdout:Sep 21 01:31:31 smithi150 podman[73327]: 2024-09-21 01:31:31.853873087 +0000 UTC m=+0.568733432 container died ebcec9f7a9c22c1295f2c16f388563a4998fbe9a60c60afe9b9cac67819d1770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2) 2024-09-21T01:31:32.065 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:31.724+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:32.065 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:31 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:31.724+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:32.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:31 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:31.882+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:32.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:31.876+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:32.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:31 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:31.876+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:32.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:32.064+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:32.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:32.064+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:32.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.4.service' 2024-09-21T01:31:32.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:32.526 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T01:31:32.526 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T01:31:32.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.5 2024-09-21T01:31:32.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:32 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:32.279+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:32.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:32 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:32.279+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:32.737 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:32 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:32.642+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:32.835 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:32.408+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:32.835 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:32.408+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:32.835 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:32 smithi160 systemd[1]: Stopping Ceph osd.5 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:32.835 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:32.834+0000 7ff652dae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:33.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:32 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:32.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:33.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:32 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:32.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:33.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:32.919+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:33.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:32.919+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:33.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:32.834+0000 7ff652dae640 -1 osd.5 136 *** Got signal Terminated *** 2024-09-21T01:31:33.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:32 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:32.834+0000 7ff652dae640 -1 osd.5 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:33.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:33 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:33.017+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:33.107 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:33 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:33.017+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:33.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:32 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:32.894+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:33.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:33 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:33.283+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:33.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:33 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:33.283+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:33.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:33 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:33.379+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:33.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:33 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:33.379+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:34.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:33 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:33.640+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:34.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:33 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:33.744+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:34.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:33 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:33.744+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:34.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:33 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:33.870+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:34.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:33 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:33.940+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:34.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:33 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:33.940+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:34.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:34.008+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:34.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:34.008+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:34.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:34.008+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.684692+0000 front 2024-09-21T01:31:08.684671+0000 (oldest deadline 2024-09-21T01:31:33.984823+0000) 2024-09-21T01:31:34.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:34 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:34.327+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:34.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:34 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:34.327+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:34.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:34.419+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:34.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:34.419+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:34.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:34.419+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:35.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:34 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:34.687+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:35.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:34 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:34.791+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:35.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:34 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:34.791+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:35.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:34 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:34.791+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:35.284 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:34 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:34.918+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:35.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:34.914+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:35.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:34.914+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:35.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:34 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:34.914+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:35.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:35.050+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:35.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:35.050+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:35.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:35.050+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.684692+0000 front 2024-09-21T01:31:08.684671+0000 (oldest deadline 2024-09-21T01:31:33.984823+0000) 2024-09-21T01:31:35.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:35.284+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:35.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:35.284+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:35.654 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:35.284+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:35.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:35.469+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:35.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:35.469+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:35.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:35.469+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:36.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:35 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:35.720+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:36.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:35 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:35.720+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:36.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:35 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:35.819+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:36.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:35 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:35.819+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:36.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:35 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:35.819+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:36.275 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:35.942+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:36.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:35.901+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:36.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:35.901+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:36.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:35 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:35.901+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:36.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:36.028+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:36.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:36.028+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:36.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:36.028+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.684692+0000 front 2024-09-21T01:31:08.684671+0000 (oldest deadline 2024-09-21T01:31:33.984823+0000) 2024-09-21T01:31:36.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:36 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:36.274+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:36.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:36 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:36.274+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:36.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:36 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:36.274+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:36.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:36.427+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:36.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:36.427+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:36.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:36.427+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:37.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:36 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:36.688+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:37.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:36 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:36.688+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:37.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:36 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:36.817+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:37.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:36 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:36.817+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:37.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:36 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:36.817+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:37.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:36 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:36.897+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:37.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:36.931+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:37.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:36.931+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:37.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:36 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:36.931+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:37.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:37.009+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.382300+0000 front 2024-09-21T01:30:55.382084+0000 (oldest deadline 2024-09-21T01:31:21.182542+0000) 2024-09-21T01:31:37.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:37.009+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.683594+0000 front 2024-09-21T01:31:01.683728+0000 (oldest deadline 2024-09-21T01:31:25.783833+0000) 2024-09-21T01:31:37.357 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5[46219]: 2024-09-21T01:31:37.009+0000 7ff64b391640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.684692+0000 front 2024-09-21T01:31:08.684671+0000 (oldest deadline 2024-09-21T01:31:33.984823+0000) 2024-09-21T01:31:37.630 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:37.420+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:37.630 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:37.420+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:37.630 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:37.420+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:37.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:37 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:37.315+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:37.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:37 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:37.315+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:37.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:37 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:37.315+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:38.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:37 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:37.689+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:38.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:37 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:37.689+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:38.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:37 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:37.817+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:38.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:37 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:37.817+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:38.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:37 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:37.817+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:38.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:37 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:37.872+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:38.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:37 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:37.872+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:38.278 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:37.890+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:38.278 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:37.890+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:38.279 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:37 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:37.890+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:38.569 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:38 smithi160 podman[74423]: 2024-09-21 01:31:38.278467955 +0000 UTC m=+5.562758226 container died 20546f6c12c17db4825355687b850ef02ea551f53d6657bba84596d2502c4512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T01:31:38.569 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:38 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:38.385+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:38.569 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:38 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:38.385+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:38.569 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:38 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:38.385+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:38.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:38 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:38.300+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:38.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:38 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:38.300+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:38.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:38 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:38.300+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:38.856 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:38 smithi160 podman[74423]: 2024-09-21 01:31:38.660144459 +0000 UTC m=+5.944434728 container cleanup 20546f6c12c17db4825355687b850ef02ea551f53d6657bba84596d2502c4512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T01:31:38.857 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:38 smithi160 bash[74423]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5 2024-09-21T01:31:39.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:38 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:38.704+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:39.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:38 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:38.704+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:39.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:38 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:38.810+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:39.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:38 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:38.810+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:39.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:38 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:38.810+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:39.258 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:38 smithi160 podman[74451]: 2024-09-21 01:31:38.928234183 +0000 UTC m=+0.651216323 container remove 20546f6c12c17db4825355687b850ef02ea551f53d6657bba84596d2502c4512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T01:31:39.259 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:38 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:38.866+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:39.259 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:38 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:38.866+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:39.259 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:38 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:38.866+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:39.271 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:38 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:38.918+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:39.271 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:38 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:38.918+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:39.531 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:39 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:39.358+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:39.531 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:39 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:39.358+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:39.532 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:39 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:39.358+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:39.532 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:39 smithi160 podman[74542]: 2024-09-21 01:31:39.161466281 +0000 UTC m=+0.015863081 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:39.532 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:39 smithi160 podman[74542]: 2024-09-21 01:31:39.279333509 +0000 UTC m=+0.133730314 container create 908af1f0c1862f8ed021c4ca8532768ccdeeb7904dd61cb6c9b14b8c6065e94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T01:31:39.532 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:39 smithi160 podman[74542]: 2024-09-21 01:31:39.531707339 +0000 UTC m=+0.386104139 container init 908af1f0c1862f8ed021c4ca8532768ccdeeb7904dd61cb6c9b14b8c6065e94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:31:39.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:39.270+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:39.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:39.270+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:39.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:39.270+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:39.858 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:39 smithi160 podman[74542]: 2024-09-21 01:31:39.535009762 +0000 UTC m=+0.389406567 container start 908af1f0c1862f8ed021c4ca8532768ccdeeb7904dd61cb6c9b14b8c6065e94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T01:31:39.858 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:39 smithi160 podman[74542]: 2024-09-21 01:31:39.623455565 +0000 UTC m=+0.477852377 container attach 908af1f0c1862f8ed021c4ca8532768ccdeeb7904dd61cb6c9b14b8c6065e94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T01:31:39.858 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 01:31:39 smithi160 podman[74542]: 2024-09-21 01:31:39.718505636 +0000 UTC m=+0.572902450 container died 908af1f0c1862f8ed021c4ca8532768ccdeeb7904dd61cb6c9b14b8c6065e94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:31:40.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:39.663+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:40.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:39.663+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:40.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:39.848+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:40.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:39.848+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:40.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:39 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:39.848+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:40.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:39 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:39.854+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:40.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:39 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:39.854+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:40.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:39 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:39.854+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:40.321 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:39.911+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:40.321 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:39.911+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:40.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.5.service' 2024-09-21T01:31:40.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:40.446 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T01:31:40.447 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T01:31:40.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.6 2024-09-21T01:31:40.606 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:40 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:40.320+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:40.606 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:40 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:40.320+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:40.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:40 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:40.320+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:40.653 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:40 smithi039 systemd[1]: Stopping Ceph osd.6 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:40.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:40.320+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:40.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:40.320+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:40.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:40.320+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:41.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:40 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:40.630+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:41.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:40 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:40.630+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:41.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:40 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:40.826+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:41.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:40 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:40.826+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:41.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:40 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:40.826+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:41.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:40 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:40.818+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:41.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:40 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:40.818+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:41.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:40 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:40.818+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:41.133 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:40.860+0000 7efe17c00640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:41.133 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:40.860+0000 7efe17c00640 -1 osd.6 136 *** Got signal Terminated *** 2024-09-21T01:31:41.133 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:40.860+0000 7efe17c00640 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:41.133 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:40.888+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:41.133 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:40 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:40.888+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:41.403 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:41 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:41.272+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:41.403 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:41 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:41.272+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:41.403 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:41 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:41.272+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:41.606 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:41.281+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:41.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:41.281+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:41.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:41.281+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:41.846 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:41 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:41.590+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:41.846 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:41 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:41.590+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:42.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:41.840+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:42.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:41.840+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:42.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:41 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:41.840+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:42.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:41 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:41.850+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:42.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:41 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:41.850+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:42.313 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:41 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:41.846+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:42.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:41 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:41.846+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:42.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:41 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:41.846+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:42.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:42.290+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:42.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:42.290+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:42.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:42.290+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:42.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:42.315+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:42.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:42.315+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:42.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:42.315+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:43.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:42 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:42.576+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:43.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:42 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:42.576+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:43.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:42 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:42.821+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:43.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:42 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:42.821+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:43.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:42 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:42.821+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:43.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:42.805+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:43.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:42.805+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:43.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:42 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:42.805+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:17.538762+0000 front 2024-09-21T01:31:17.538809+0000 (oldest deadline 2024-09-21T01:31:42.238884+0000) 2024-09-21T01:31:43.296 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:42.865+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:43.296 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:42.865+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:43.296 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:42 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:42.865+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:43.606 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:43.295+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:43.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:43.295+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:43.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:43.295+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:43.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:43.295+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:43.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:43 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:43.342+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:43.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:43 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:43.342+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:43.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:43 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:43.342+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:44.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:43.596+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:44.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:43.596+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:44.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:43.805+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:44.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:43.805+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:44.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:43 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:43.805+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:44.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:43 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:43.780+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:44.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:43 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:43.780+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:44.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:43 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:43.780+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:17.538762+0000 front 2024-09-21T01:31:17.538809+0000 (oldest deadline 2024-09-21T01:31:42.238884+0000) 2024-09-21T01:31:44.341 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:43.906+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:44.342 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:43.906+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:44.342 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:43 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:43.906+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:44.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:44.341+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:44.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:44.341+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:44.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:44.341+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:44.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:44.341+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:44.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:44 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:44.331+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:44.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:44 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:44.331+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:44.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:44 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:44.331+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:45.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:44 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:44.586+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:45.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:44 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:44.586+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:45.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:44 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:44.765+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:45.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:44 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:44.765+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:45.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:44 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:44.765+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:45.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:44 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:44.761+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:45.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:44 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:44.761+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:45.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:44 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:44.761+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:17.538762+0000 front 2024-09-21T01:31:17.538809+0000 (oldest deadline 2024-09-21T01:31:42.238884+0000) 2024-09-21T01:31:45.339 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:44.865+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:45.340 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:44.865+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:45.340 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:44 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:44.865+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:45.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:45.339+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:45.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:45.339+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:45.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:45.339+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:45.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:45.339+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:45.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:45.365+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:45.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:45.365+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:45.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:45.365+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:45.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:45.365+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:46.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:45 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:45.585+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:46.064 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:45 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:45.585+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:46.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:45 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:45.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:46.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:45 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:45.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:46.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:45 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:45.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:46.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:45 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:45.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:46.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:45.840+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:46.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:45.840+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:46.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:45 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:45.840+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:46.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:45.743+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.135145+0000 front 2024-09-21T01:31:04.135077+0000 (oldest deadline 2024-09-21T01:31:26.435196+0000) 2024-09-21T01:31:46.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:45.743+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.136971+0000 front 2024-09-21T01:31:08.137061+0000 (oldest deadline 2024-09-21T01:31:37.538074+0000) 2024-09-21T01:31:46.153 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6[59446]: 2024-09-21T01:31:45.743+0000 7efe0e600640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:17.538762+0000 front 2024-09-21T01:31:17.538809+0000 (oldest deadline 2024-09-21T01:31:42.238884+0000) 2024-09-21T01:31:46.606 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:46.354+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:46.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:46.354+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:46.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:46.354+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:46.607 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:46.354+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:46.752 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:46 smithi039 podman[108243]: 2024-09-21 01:31:46.447408499 +0000 UTC m=+5.701487584 container died 60f525769fef034247fae276d7af75214176e1739fb5cad9db1196c491239ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T01:31:46.752 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:46.413+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:46.753 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:46.413+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:46.753 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:46.413+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:46.753 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:46 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:46.413+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:46.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:46 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:46.542+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:46.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:46 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:46.542+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:46.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:46 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:46.770+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:46.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:46 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:46.770+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:46.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:46 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:46.770+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:46.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:46 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:46.770+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:47.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:46.867+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:47.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:46.867+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:47.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:46 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:46.867+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:47.403 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:47 smithi039 podman[108243]: 2024-09-21 01:31:47.108252083 +0000 UTC m=+6.362331164 container cleanup 60f525769fef034247fae276d7af75214176e1739fb5cad9db1196c491239ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:31:47.403 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:47 smithi039 bash[108243]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6 2024-09-21T01:31:47.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:47.558+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:47.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:47.558+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:47.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:47.779+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:47.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:47.779+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:47.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:47.779+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:47.814 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:47.779+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:47.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:47.402+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:47.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:47.402+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:47.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:47.402+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:47.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:47.402+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:47.884 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:47 smithi039 podman[108271]: 2024-09-21 01:31:47.443427916 +0000 UTC m=+0.986740033 container remove 60f525769fef034247fae276d7af75214176e1739fb5cad9db1196c491239ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:31:47.884 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:47 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:47.438+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:47.884 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:47 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:47.438+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:47.884 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:47 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:47.438+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:47.885 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:47 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:47.438+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:48.137 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:47 smithi039 podman[108360]: 2024-09-21 01:31:47.788974672 +0000 UTC m=+0.027753624 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:48.137 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:47 smithi039 podman[108360]: 2024-09-21 01:31:47.900408932 +0000 UTC m=+0.139187883 container create 459f9a9f72f4a8b2db5514ad91b22b9b379eee5c64f0707090f782126d14686f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T01:31:48.137 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:48 smithi039 podman[108360]: 2024-09-21 01:31:48.130384849 +0000 UTC m=+0.369163803 container init 459f9a9f72f4a8b2db5514ad91b22b9b379eee5c64f0707090f782126d14686f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T01:31:48.137 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:48 smithi039 podman[108360]: 2024-09-21 01:31:48.137151337 +0000 UTC m=+0.375930300 container start 459f9a9f72f4a8b2db5514ad91b22b9b379eee5c64f0707090f782126d14686f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T01:31:48.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:47.908+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:48.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:47.908+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:48.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:47 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:47.908+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:48.403 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:48 smithi039 podman[108360]: 2024-09-21 01:31:48.230083715 +0000 UTC m=+0.468862678 container attach 459f9a9f72f4a8b2db5514ad91b22b9b379eee5c64f0707090f782126d14686f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:31:48.696 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:48.432+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:48.696 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:48.432+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:48.696 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:48.432+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:48.696 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:48.432+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:48.797 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:48.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:48.797 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:48.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:48.797 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:48.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:48.874 INFO:journalctl@ceph.osd.6.smithi039.stdout:Sep 21 01:31:48 smithi039 podman[108360]: 2024-09-21 01:31:48.408128974 +0000 UTC m=+0.646907934 container died 459f9a9f72f4a8b2db5514ad91b22b9b379eee5c64f0707090f782126d14686f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T01:31:48.874 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:48 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:48.470+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:48.874 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:48 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:48.470+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:48.875 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:48 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:48.470+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:48.875 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:48 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:48.470+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:49.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:48.795+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:49.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:48.795+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:49.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:48.795+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:49.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:48 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:48.795+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:49.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.6.service' 2024-09-21T01:31:49.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:49.300 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T01:31:49.300 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T01:31:49.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.7 2024-09-21T01:31:49.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:48.950+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:49.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:48.950+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:49.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:48 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:48.950+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 systemd[1]: Stopping Ceph osd.7 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:49.516+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:49.516+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:49.516+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:49.710+0000 7f227ed0e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:49.710+0000 7f227ed0e640 -1 osd.7 136 *** Got signal Terminated *** 2024-09-21T01:31:49.712 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:49.710+0000 7f227ed0e640 -1 osd.7 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:49.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:49.399+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:49.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:49.399+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:49.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:49.399+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:49.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:49.399+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:49.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:49 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:49.513+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:49.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:49 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:49.513+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:49.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:49 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:49.513+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:49.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:49 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:49.513+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:50.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:49.783+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:50.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:49.783+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:50.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:49.783+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:50.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:49 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:49.783+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:50.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:49.984+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:50.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:49.984+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:50.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:49 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:49.984+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:50.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:50.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:50.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:50.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:50.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:50.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:50.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:50 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:50.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:50.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:50 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:50.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:50.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:50 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:50.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:50.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:50 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:50.412+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:50.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:50 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:50.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:50.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:50 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:50.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:50.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:50 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:50.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:50.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:50 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:50.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:51.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:50.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:51.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:50.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:51.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:50.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:51.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:50 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:50.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:51.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.022+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:51.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.022+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:51.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.022+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:51.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.022+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:51.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:51.550+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:51.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:51.550+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:51.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:51.550+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:51.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:51.550+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.346840+0000 front 2024-09-21T01:31:28.346819+0000 (oldest deadline 2024-09-21T01:31:50.547863+0000) 2024-09-21T01:31:51.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:51.429+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:51.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:51.429+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:51.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:51.429+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:51.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:51.429+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:51.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:51.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:51.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:51.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:51.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:51.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:51.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:51 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:51.506+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:52.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:51.824+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:52.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:51.824+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:52.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:51.824+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:52.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:51 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:51.824+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:52.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.981+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:52.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.981+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:52.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.981+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:52.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:51 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:51.981+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:52.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:52.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:52.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:52.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:52.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:52.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:52.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:52.524+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.346840+0000 front 2024-09-21T01:31:28.346819+0000 (oldest deadline 2024-09-21T01:31:50.547863+0000) 2024-09-21T01:31:52.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:52.479+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:52.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:52.479+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:52.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:52.479+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:52.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:52.479+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:52.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:52.530+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:52.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:52.530+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:52.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:52.530+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:52.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:52.530+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:52.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:52 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:52.530+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:53.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:52.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:53.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:52.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:53.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:52.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:53.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:52.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:53.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:52.990+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:53.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:52.990+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:53.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:52.990+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:53.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:52 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:52.990+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:53.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:53.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:53.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:53.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:53.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:53.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:53.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:53.530+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.346840+0000 front 2024-09-21T01:31:28.346819+0000 (oldest deadline 2024-09-21T01:31:50.547863+0000) 2024-09-21T01:31:53.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:53 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:53.453+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:53.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:53 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:53.453+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:53.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:53 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:53.453+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:53.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:53 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:53.453+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:53.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:53 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:53.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:53.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:53 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:53.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:53.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:53 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:53.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:53.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:53 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:53.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:53.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:53 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:53.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:54.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:53.849+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:54.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:53.849+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:54.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:53.849+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:54.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:53.849+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:54.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:53 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:53.849+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:31:54.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:54.013+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:54.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:54.013+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:54.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:54.013+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:54.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:54.013+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:54.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:54.483+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:04.041004+0000 front 2024-09-21T01:31:04.041033+0000 (oldest deadline 2024-09-21T01:31:26.941184+0000) 2024-09-21T01:31:54.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:54.483+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.742834+0000 front 2024-09-21T01:31:12.742964+0000 (oldest deadline 2024-09-21T01:31:35.042904+0000) 2024-09-21T01:31:54.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:54.483+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.845250+0000 front 2024-09-21T01:31:20.845111+0000 (oldest deadline 2024-09-21T01:31:47.845797+0000) 2024-09-21T01:31:54.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7[52449]: 2024-09-21T01:31:54.483+0000 7f22772f1640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.346840+0000 front 2024-09-21T01:31:28.346819+0000 (oldest deadline 2024-09-21T01:31:50.547863+0000) 2024-09-21T01:31:54.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:54.482+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:54.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:54.482+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:54.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:54.482+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:54.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:54 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:54.482+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:54.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:54 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:54.543+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:54.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:54 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:54.543+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:54.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:54 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:54.543+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:54.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:54 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:54.543+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:54.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:54 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:54.543+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:55.255 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:54.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:55.255 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:54.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:55.255 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:54.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:55.255 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:54.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:55.256 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:54 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:54.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:31:55.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.034+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:55.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.034+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:55.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.034+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:55.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.034+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:55.554 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:55 smithi150 podman[73430]: 2024-09-21 01:31:55.255432894 +0000 UTC m=+5.753557804 container died 1ad3a6744966f09e7676e561f1b7ced926d6128c0f4a7d501e88c70a5c59ffea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:31:55.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:55 smithi150 podman[73430]: 2024-09-21 01:31:55.627907805 +0000 UTC m=+6.126032715 container cleanup 1ad3a6744966f09e7676e561f1b7ced926d6128c0f4a7d501e88c70a5c59ffea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:31:55.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:55 smithi150 bash[73430]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7 2024-09-21T01:31:55.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:55.450+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:55.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:55.450+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:55.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:55.450+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:55.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:55.450+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:55.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:55 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:55.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:55.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:55 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:55.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:55.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:55 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:55.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:55.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:55 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:55.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:55.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:55 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:55.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:56.244 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:55 smithi150 podman[73445]: 2024-09-21 01:31:55.904324983 +0000 UTC m=+0.644742520 container remove 1ad3a6744966f09e7676e561f1b7ced926d6128c0f4a7d501e88c70a5c59ffea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T01:31:56.244 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:55 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:55.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:56.245 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:55 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:55.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:56.245 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:55 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:55.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:56.245 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:55 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:55.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:56.245 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:55 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:55.825+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:31:56.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.988+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:56.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.989+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:56.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.989+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:56.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:55 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:55.989+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:56.518 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:56 smithi150 podman[73535]: 2024-09-21 01:31:56.147036616 +0000 UTC m=+0.015880967 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:31:56.518 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:56 smithi150 podman[73535]: 2024-09-21 01:31:56.265567869 +0000 UTC m=+0.134412211 container create 21be28fba855825a7324cf3c01f3f1c02c25cdacd4204ca470fb214476b055ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:31:56.518 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:56 smithi150 podman[73535]: 2024-09-21 01:31:56.516003768 +0000 UTC m=+0.384848111 container init 21be28fba855825a7324cf3c01f3f1c02c25cdacd4204ca470fb214476b055ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T01:31:56.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:56 smithi150 podman[73535]: 2024-09-21 01:31:56.519495506 +0000 UTC m=+0.388339849 container start 21be28fba855825a7324cf3c01f3f1c02c25cdacd4204ca470fb214476b055ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:31:56.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:56 smithi150 podman[73535]: 2024-09-21 01:31:56.599136414 +0000 UTC m=+0.467980780 container attach 21be28fba855825a7324cf3c01f3f1c02c25cdacd4204ca470fb214476b055ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:31:56.814 INFO:journalctl@ceph.osd.7.smithi150.stdout:Sep 21 01:31:56 smithi150 podman[73535]: 2024-09-21 01:31:56.7114053 +0000 UTC m=+0.580249655 container died 21be28fba855825a7324cf3c01f3f1c02c25cdacd4204ca470fb214476b055ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T01:31:56.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:56.494+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:56.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:56.494+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:56.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:56.494+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:56.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:56.494+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:56.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:56 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:56.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:56.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:56 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:56.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:56.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:56 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:56.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:56.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:56 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:56.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:56.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:56 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:56.499+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:57.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:56 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:56.875+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:57.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:56 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:56.875+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:57.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:56 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:56.875+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:57.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:56 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:56.875+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:57.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:56 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:56.875+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:31:57.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:56.980+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:57.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:56.980+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:57.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:56.980+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:57.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:56 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:56.980+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:57.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.7.service' 2024-09-21T01:31:57.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:31:57.427 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T01:31:57.428 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-21T01:31:57.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.8 2024-09-21T01:31:57.797 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.461+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:57.797 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.461+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:57.797 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.461+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:57.797 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.461+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:57.797 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 systemd[1]: Stopping Ceph osd.8 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:31:57.797 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.795+0000 7f6021b3d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:31:57.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:57.477+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:57.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:57.477+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:57.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:57.477+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:57.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:57.477+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:57.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:57 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:57.477+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:58.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:57.835+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:58.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:57.835+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:58.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:57.835+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:58.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:57.835+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:58.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:57.835+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:31:58.106 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.795+0000 7f6021b3d640 -1 osd.8 136 *** Got signal Terminated *** 2024-09-21T01:31:58.107 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:57 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:57.795+0000 7f6021b3d640 -1 osd.8 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:31:58.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.025+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:58.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.025+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:58.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.025+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:58.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.025+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:58.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:58.478+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:58.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:58.478+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:58.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:58.478+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:58.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:58.478+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:58.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:58 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:58.516+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:58.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:58 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:58.516+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:58.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:58 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:58.516+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:58.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:58 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:58.516+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:58.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:58 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:58.516+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:31:59.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:58 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:58.847+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:31:59.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:58 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:58.847+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:31:59.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:58 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:58.847+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:31:59.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:58 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:58.847+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:31:59.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:58 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:58.847+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:31:59.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.992+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:31:59.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.992+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:31:59.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.992+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:31:59.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.992+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:31:59.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:31:58 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:31:58.992+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:31:59.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:59 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:59.477+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:31:59.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:59 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:59.477+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:31:59.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:59 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:59.477+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:31:59.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:31:59 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:31:59.477+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:31:59.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:59 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:59.518+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:31:59.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:59 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:59.518+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:31:59.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:59 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:59.518+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:31:59.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:59 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:59.518+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:31:59.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:31:59 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:31:59.518+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:00.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:59.827+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:00.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:59.827+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:00.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:59.827+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:00.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:59.827+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:00.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:31:59 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:31:59.827+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:00.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:00.012+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:00.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:00.012+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:00.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:00.012+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:00.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:00.012+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:00.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:00.012+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:00.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:00.472+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:32:00.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:00.472+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:32:00.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:00.472+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:32:00.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:00 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:00.472+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:32:00.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:00.544+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:00.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:00.544+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:00.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:00.544+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:00.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:00.544+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:00.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:00.544+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:00.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:00 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:00.544+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:01.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:00.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:01.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:00.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:01.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:00.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:01.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:00.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:01.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:00.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:01.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:01.038+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:01.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:01.038+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:01.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:01.038+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:01.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:01.038+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:01.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:01.038+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:01.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:01.459+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:32:01.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:01.459+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:32:01.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:01.459+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:32:01.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:01 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:01.459+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:32:01.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:01.547+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:01.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:01.547+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:01.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:01.547+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:01.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:01.547+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:01.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:01.547+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:01.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:01 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:01.547+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:02.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:01 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:01.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:02.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:01 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:01.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:02.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:01 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:01.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:02.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:01 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:01.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:02.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:01 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:01.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:02.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:01 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:01.873+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:02.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:02.060+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:02.356 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:02.060+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:02.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:02.060+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:02.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:02.060+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:02.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:02.060+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:02.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:02.462+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:55.904851+0000 front 2024-09-21T01:30:55.904780+0000 (oldest deadline 2024-09-21T01:31:18.804718+0000) 2024-09-21T01:32:02.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:02.462+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:01.705794+0000 front 2024-09-21T01:31:01.705831+0000 (oldest deadline 2024-09-21T01:31:26.306715+0000) 2024-09-21T01:32:02.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:02.462+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.207805+0000 front 2024-09-21T01:31:12.207746+0000 (oldest deadline 2024-09-21T01:31:33.907959+0000) 2024-09-21T01:32:02.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:02.462+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.210233+0000 front 2024-09-21T01:31:20.210040+0000 (oldest deadline 2024-09-21T01:31:42.510114+0000) 2024-09-21T01:32:02.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:02 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8[52716]: 2024-09-21T01:32:02.462+0000 7f601e128640 -1 osd.8 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:32.313003+0000 front 2024-09-21T01:31:32.313024+0000 (oldest deadline 2024-09-21T01:32:02.313615+0000) 2024-09-21T01:32:02.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:02.569+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:02.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:02.569+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:02.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:02.569+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:02.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:02.569+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:02.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:02.569+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:02.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:02 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:02.569+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:03.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:02 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:02.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:03.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:02 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:02.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:03.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:02 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:02.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:03.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:02 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:02.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:03.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:02 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:02.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:03.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:02 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:02.852+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:03.356 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:03 smithi160 podman[74644]: 2024-09-21 01:32:03.284515706 +0000 UTC m=+5.654926549 container died bd24e72e0145f95558ce5f257dd22030ba8a9be8c6a613e6873a8a6d914167c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T01:32:03.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:03.086+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:03.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:03.086+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:03.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:03.086+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:03.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:03.086+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:03.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:03.086+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:03.856 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:03 smithi160 podman[74644]: 2024-09-21 01:32:03.637913635 +0000 UTC m=+6.008324473 container cleanup bd24e72e0145f95558ce5f257dd22030ba8a9be8c6a613e6873a8a6d914167c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T01:32:03.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:03 smithi160 bash[74644]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8 2024-09-21T01:32:03.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:03 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:03.598+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:03.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:03 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:03.598+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:03.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:03 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:03.598+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:03.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:03 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:03.598+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:03.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:03 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:03.598+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:03.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:03 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:03.598+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:04.257 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:03 smithi160 podman[74659]: 2024-09-21 01:32:03.929208847 +0000 UTC m=+0.641296189 container remove bd24e72e0145f95558ce5f257dd22030ba8a9be8c6a613e6873a8a6d914167c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T01:32:04.258 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:04 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:04.061+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:04.258 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:04 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:04.061+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:04.258 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:04 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:04.061+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:04.258 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:04 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:04.061+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:04.258 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:04 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:04.061+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:04.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:03 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:03.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:04.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:03 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:03.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:04.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:03 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:03.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:04.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:03 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:03.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:04.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:03 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:03.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:04.314 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:03 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:03.815+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:04.525 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:04 smithi160 podman[74749]: 2024-09-21 01:32:04.160503712 +0000 UTC m=+0.015896532 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:32:04.526 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:04 smithi160 podman[74749]: 2024-09-21 01:32:04.28170158 +0000 UTC m=+0.137094393 container create 5dc5bcba0b6e12d9f6739dec4b1fb7169fa665d4e49f7f1466f5981015687025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:32:04.526 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:04 smithi160 podman[74749]: 2024-09-21 01:32:04.525929474 +0000 UTC m=+0.381322292 container init 5dc5bcba0b6e12d9f6739dec4b1fb7169fa665d4e49f7f1466f5981015687025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T01:32:04.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:04 smithi160 podman[74749]: 2024-09-21 01:32:04.529291291 +0000 UTC m=+0.384684105 container start 5dc5bcba0b6e12d9f6739dec4b1fb7169fa665d4e49f7f1466f5981015687025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0) 2024-09-21T01:32:04.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:04 smithi160 podman[74749]: 2024-09-21 01:32:04.600984701 +0000 UTC m=+0.456377522 container attach 5dc5bcba0b6e12d9f6739dec4b1fb7169fa665d4e49f7f1466f5981015687025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T01:32:04.857 INFO:journalctl@ceph.osd.8.smithi160.stdout:Sep 21 01:32:04 smithi160 podman[74749]: 2024-09-21 01:32:04.713132408 +0000 UTC m=+0.568525222 container died 5dc5bcba0b6e12d9f6739dec4b1fb7169fa665d4e49f7f1466f5981015687025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-8-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:32:04.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:04 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:04.566+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:04.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:04 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:04.566+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:04.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:04 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:04.566+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:04.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:04 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:04.566+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:04.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:04 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:04.566+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:04.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:04 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:04.566+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:05.063 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:04 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:04.771+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:05.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:04 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:04.771+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:05.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:04 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:04.771+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:05.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:04 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:04.771+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:05.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:04 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:04.771+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:05.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:04 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:04.771+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:05.271 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:05 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:05.065+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:05.271 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:05 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:05.065+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:05.271 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:05 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:05.065+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:05.271 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:05 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:05.065+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:05.271 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:05 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:05.065+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:05.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.8.service' 2024-09-21T01:32:05.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:05.333 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-21T01:32:05.333 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-21T01:32:05.333 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.9 2024-09-21T01:32:05.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 systemd[1]: Stopping Ceph osd.9 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:05.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.592+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:05.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.592+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:05.653 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.592+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:05.654 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.592+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:05.654 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.592+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:05.654 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.592+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:06.065 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:05 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:05.784+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:06.066 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:05 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:05.784+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:06.066 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:05 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:05.784+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:06.066 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:05 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:05.784+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:06.066 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:05 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:05.784+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:06.066 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:05 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:05.784+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:06.153 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.746+0000 7ff95ee00640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:06.153 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.746+0000 7ff95ee00640 -1 osd.9 136 *** Got signal Terminated *** 2024-09-21T01:32:06.153 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:05 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:05.746+0000 7ff95ee00640 -1 osd.9 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:32:06.497 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:06 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:06.108+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:06.497 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:06 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:06.108+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:06.497 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:06 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:06.108+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:06.497 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:06 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:06.108+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:06.497 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:06 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:06.108+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:06.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:06 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:06.564+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:06.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:06 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:06.564+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:06.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:06 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:06.564+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:06.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:06 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:06.564+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:06.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:06 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:06.564+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:06.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:06 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:06.564+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:07.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:06 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:06.750+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:07.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:06 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:06.750+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:07.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:06 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:06.750+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:07.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:06 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:06.750+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:07.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:06 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:06.750+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:07.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:06 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:06.750+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:07.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:07 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:07.071+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:07.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:07 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:07.071+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:07.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:07 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:07.071+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:07.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:07 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:07.071+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:07.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:07 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:07.071+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:07.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:07.575+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:07.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:07.575+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:07.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:07.575+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:07.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:07.575+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:07.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:07.575+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:07.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:07 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:07.575+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:08.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:07 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:07.776+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:08.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:07 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:07.776+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:08.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:07 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:07.776+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:08.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:07 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:07.776+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:08.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:07 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:07.776+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:08.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:07 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:07.776+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:08.369 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:08.107+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:08.369 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:08.107+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:08.369 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:08.107+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:08.369 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:08.107+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:08.369 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:08.107+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:08.369 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:08.107+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:08.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:08 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:08.576+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:08.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:08 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:08.576+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:08.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:08 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:08.576+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:08.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:08 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:08.576+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:08.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:08 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:08.576+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:08.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:08 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:08.576+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:09.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:08.752+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:09.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:08.752+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:09.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:08.752+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:09.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:08.752+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:09.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:08.752+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:09.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:08 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:08.752+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:09.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:09 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:09.085+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:09.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:09 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:09.085+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:09.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:09 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:09.085+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:09.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:09 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:09.085+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:09.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:09 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:09.085+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:09.357 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:09 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:09.085+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:09.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:09 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:09.570+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:09.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:09 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:09.570+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:09.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:09 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:09.570+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:09.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:09 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:09.570+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:09.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:09 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:09.570+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:09.904 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:09 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:09.570+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:10.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:09 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:09.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:10.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:09 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:09.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:10.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:09 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:09.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:10.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:09 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:09.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:10.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:09 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:09.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:10.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:09 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:09.725+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:10.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:10 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:10.127+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:10.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:10 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:10.127+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:10.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:10 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:10.127+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:10.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:10 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:10.127+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:10.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:10 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:10.127+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:10.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:10 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:10.127+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:10.805 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:10 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:10.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:53.018193+0000 front 2024-09-21T01:30:53.018231+0000 (oldest deadline 2024-09-21T01:31:17.718366+0000) 2024-09-21T01:32:10.805 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:10 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:10.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.421902+0000 front 2024-09-21T01:31:03.421866+0000 (oldest deadline 2024-09-21T01:31:28.722075+0000) 2024-09-21T01:32:10.805 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:10 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:10.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:08.722767+0000 front 2024-09-21T01:31:08.722799+0000 (oldest deadline 2024-09-21T01:31:34.622965+0000) 2024-09-21T01:32:10.805 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:10 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:10.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:18.625236+0000 front 2024-09-21T01:31:18.625206+0000 (oldest deadline 2024-09-21T01:31:44.525407+0000) 2024-09-21T01:32:10.805 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:10 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:10.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:28.026486+0000 front 2024-09-21T01:31:28.026755+0000 (oldest deadline 2024-09-21T01:31:51.526691+0000) 2024-09-21T01:32:10.805 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:10 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9[65991]: 2024-09-21T01:32:10.531+0000 7ff95a800640 -1 osd.9 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:37.427895+0000 front 2024-09-21T01:31:37.427718+0000 (oldest deadline 2024-09-21T01:31:59.727585+0000) 2024-09-21T01:32:11.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:10 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:10.743+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:11.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:10 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:10.743+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:11.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:10 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:10.743+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:11.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:10 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:10.743+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:11.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:10 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:10.743+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:11.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:10 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:10.743+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:11.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:11.175+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:11.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:11.175+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:11.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:11.175+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:11.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:11.175+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:11.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:11.175+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:11.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:11 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:11.175+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:11.646 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:11 smithi039 podman[108463]: 2024-09-21 01:32:11.3373531 +0000 UTC m=+5.719516518 container died 2dda4214b92309511b493ab81eb1ce8b5f52145b398e8e7a820e1f25337b9433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:32:11.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:11 smithi039 podman[108463]: 2024-09-21 01:32:11.77748188 +0000 UTC m=+6.159645280 container cleanup 2dda4214b92309511b493ab81eb1ce8b5f52145b398e8e7a820e1f25337b9433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T01:32:11.903 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:11 smithi039 bash[108463]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:12.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:11 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:11.754+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:12.403 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:12 smithi039 podman[108481]: 2024-09-21 01:32:12.137178268 +0000 UTC m=+0.792699851 container remove 2dda4214b92309511b493ab81eb1ce8b5f52145b398e8e7a820e1f25337b9433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:32:12.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:12.212+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:12.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:12.212+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:12.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:12.212+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:12.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:12.212+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:12.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:12.212+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:12.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:12.212+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:13.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:12 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:12.731+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:13.098 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:12 smithi039 podman[108578]: 2024-09-21 01:32:12.645758605 +0000 UTC m=+0.027846635 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:32:13.098 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:12 smithi039 podman[108578]: 2024-09-21 01:32:12.754453509 +0000 UTC m=+0.136541527 container create 29b0eee1116b2f2befd763bcb49f4694a8492c547abce92210046e1eb211c06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:32:13.098 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:13 smithi039 podman[108578]: 2024-09-21 01:32:13.098488251 +0000 UTC m=+0.480576298 container init 29b0eee1116b2f2befd763bcb49f4694a8492c547abce92210046e1eb211c06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:32:13.381 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:13 smithi039 podman[108578]: 2024-09-21 01:32:13.105986767 +0000 UTC m=+0.488074786 container start 29b0eee1116b2f2befd763bcb49f4694a8492c547abce92210046e1eb211c06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T01:32:13.381 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:13 smithi039 podman[108578]: 2024-09-21 01:32:13.207077491 +0000 UTC m=+0.589165532 container attach 29b0eee1116b2f2befd763bcb49f4694a8492c547abce92210046e1eb211c06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:32:13.381 INFO:journalctl@ceph.osd.9.smithi039.stdout:Sep 21 01:32:13 smithi039 podman[108578]: 2024-09-21 01:32:13.3809278 +0000 UTC m=+0.763015824 container died 29b0eee1116b2f2befd763bcb49f4694a8492c547abce92210046e1eb211c06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-9-deactivate, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T01:32:13.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:13 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:13.216+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:13.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:13 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:13.216+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:13.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:13 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:13.216+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:13.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:13 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:13.216+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:13.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:13 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:13.216+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:13.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:13 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:13.216+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:14.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.9.service' 2024-09-21T01:32:14.063 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:14.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:14.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:14.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:14.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:14.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:14.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:13 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:13.711+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:14.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:14.112 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-21T01:32:14.112 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-21T01:32:14.113 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.10 2024-09-21T01:32:14.439 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 systemd[1]: Stopping Ceph osd.10 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:14.606 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:14 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:14.187+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:14.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:14 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:14.187+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:14.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:14 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:14.187+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:14.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:14 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:14.187+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:14.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:14 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:14.187+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:14.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:14 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:14.187+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:14.717 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.438+0000 7fe150aaa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:14.718 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.438+0000 7fe150aaa640 -1 osd.10 136 *** Got signal Terminated *** 2024-09-21T01:32:14.718 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.438+0000 7fe150aaa640 -1 osd.10 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:15.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:14 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:14.717+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:15.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:15 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:15.192+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:15.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:15 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:15.192+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:15.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:15 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:15.192+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:15.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:15 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:15.192+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:15.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:15 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:15.192+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:15.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:15 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:15.192+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:16.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:15 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:15.737+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:16.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:16.189+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:17.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:16 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:16.735+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:17.610 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:17.610 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:17.610 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:17.610 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:17.610 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:17.611 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:17.611 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:17 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:17.207+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:18.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:17 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:17.726+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:18.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:18 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:18.231+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:52.242685+0000 front 2024-09-21T01:30:52.242742+0000 (oldest deadline 2024-09-21T01:31:16.942840+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:02.244069+0000 front 2024-09-21T01:31:02.244090+0000 (oldest deadline 2024-09-21T01:31:27.544263+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:11.645331+0000 front 2024-09-21T01:31:11.645532+0000 (oldest deadline 2024-09-21T01:31:34.545381+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-09-21T01:31:20.446635+0000 front 2024-09-21T01:31:20.446616+0000 (oldest deadline 2024-09-21T01:31:45.146709+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.847633+0000 front 2024-09-21T01:31:29.847605+0000 (oldest deadline 2024-09-21T01:31:53.347783+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:35.048885+0000 front 2024-09-21T01:31:35.048845+0000 (oldest deadline 2024-09-21T01:32:00.948803+0000) 2024-09-21T01:32:19.064 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:18 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10[59152]: 2024-09-21T01:32:18.758+0000 7fe14d095640 -1 osd.10 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:45.450868+0000 front 2024-09-21T01:31:45.450969+0000 (oldest deadline 2024-09-21T01:32:10.750899+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:19.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:19 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:19.195+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:20.256 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:19 smithi150 podman[73651]: 2024-09-21 01:32:19.974600013 +0000 UTC m=+5.658936677 container died c21126c066440fff75f9f330c5f3d3a80f3fc6f5cede077da0ec7ba2a5491b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T01:32:20.563 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:20 smithi150 podman[73651]: 2024-09-21 01:32:20.36901569 +0000 UTC m=+6.053352342 container cleanup c21126c066440fff75f9f330c5f3d3a80f3fc6f5cede077da0ec7ba2a5491b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T01:32:20.564 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:20 smithi150 bash[73651]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:20.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:20 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:20.172+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:20.965 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:20 smithi150 podman[73668]: 2024-09-21 01:32:20.656703705 +0000 UTC m=+0.671476520 container remove c21126c066440fff75f9f330c5f3d3a80f3fc6f5cede077da0ec7ba2a5491b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T01:32:21.268 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:20 smithi150 podman[73756]: 2024-09-21 01:32:20.965849836 +0000 UTC m=+0.112754549 container create eba5ca6f6fee1a0303ef92e0dfeb68840fbc9a3fa793999b80fc29a84a4e02b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:32:21.268 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:20 smithi150 podman[73756]: 2024-09-21 01:32:20.868493705 +0000 UTC m=+0.015398417 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:32:21.268 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:21 smithi150 podman[73756]: 2024-09-21 01:32:21.201585048 +0000 UTC m=+0.348489760 container init eba5ca6f6fee1a0303ef92e0dfeb68840fbc9a3fa793999b80fc29a84a4e02b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T01:32:21.268 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:21 smithi150 podman[73756]: 2024-09-21 01:32:21.204785267 +0000 UTC m=+0.351689989 container start eba5ca6f6fee1a0303ef92e0dfeb68840fbc9a3fa793999b80fc29a84a4e02b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2) 2024-09-21T01:32:21.269 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:21 smithi150 podman[73756]: 2024-09-21 01:32:21.268142949 +0000 UTC m=+0.415047670 container attach eba5ca6f6fee1a0303ef92e0dfeb68840fbc9a3fa793999b80fc29a84a4e02b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:32:21.564 INFO:journalctl@ceph.osd.10.smithi150.stdout:Sep 21 01:32:21 smithi150 podman[73756]: 2024-09-21 01:32:21.388212104 +0000 UTC m=+0.535116816 container died eba5ca6f6fee1a0303ef92e0dfeb68840fbc9a3fa793999b80fc29a84a4e02b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:21.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:21 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:21.210+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:22.067 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.10.service' 2024-09-21T01:32:22.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:22.108 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-21T01:32:22.108 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-21T01:32:22.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.11 2024-09-21T01:32:22.449 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 systemd[1]: Stopping Ceph osd.11 for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:22.449 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:22.449 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:22.449 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:22.449 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:22.449 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:22.450 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:22.450 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.222+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:22.450 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.448+0000 7fe958c5e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:22.450 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.448+0000 7fe958c5e640 -1 osd.11 136 *** Got signal Terminated *** 2024-09-21T01:32:22.450 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:22 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:22.448+0000 7fe958c5e640 -1 osd.11 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:23.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:23 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:23.177+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:24.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:24 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:24.214+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:25.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:25 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:25.208+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:26.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:26 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:26.166+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6820 osd.8 since back 2024-09-21T01:32:01.582244+0000 front 2024-09-21T01:32:01.582287+0000 (oldest deadline 2024-09-21T01:32:25.682301+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-09-21T01:30:54.269655+0000 front 2024-09-21T01:30:54.269482+0000 (oldest deadline 2024-09-21T01:31:18.369678+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-09-21T01:31:03.671294+0000 front 2024-09-21T01:31:03.670998+0000 (oldest deadline 2024-09-21T01:31:27.171190+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6804 osd.2 since back 2024-09-21T01:31:12.973137+0000 front 2024-09-21T01:31:12.973099+0000 (oldest deadline 2024-09-21T01:31:34.073272+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-09-21T01:31:29.876084+0000 front 2024-09-21T01:31:29.876244+0000 (oldest deadline 2024-09-21T01:31:50.976056+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6812 osd.5 since back 2024-09-21T01:31:36.077943+0000 front 2024-09-21T01:31:36.078036+0000 (oldest deadline 2024-09-21T01:31:58.377940+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2024-09-21T01:31:42.479341+0000 front 2024-09-21T01:31:42.479325+0000 (oldest deadline 2024-09-21T01:32:07.179135+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-09-21T01:31:53.580782+0000 front 2024-09-21T01:31:53.580664+0000 (oldest deadline 2024-09-21T01:32:15.280531+0000) 2024-09-21T01:32:27.607 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11[59123]: 2024-09-21T01:32:27.190+0000 7fe955249640 -1 osd.11 136 heartbeat_check: no reply from 172.21.15.160:6820 osd.8 since back 2024-09-21T01:32:01.582244+0000 front 2024-09-21T01:32:01.582287+0000 (oldest deadline 2024-09-21T01:32:25.682301+0000) 2024-09-21T01:32:28.106 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:27 smithi160 podman[74864]: 2024-09-21 01:32:27.820270372 +0000 UTC m=+5.513760229 container died 2d5f5901e912bb140ec7860e517b34ca0c7c219d07e2a6c3927500122360eb86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T01:32:28.457 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:28 smithi160 podman[74864]: 2024-09-21 01:32:28.189956465 +0000 UTC m=+5.883446309 container cleanup 2d5f5901e912bb140ec7860e517b34ca0c7c219d07e2a6c3927500122360eb86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T01:32:28.458 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:28 smithi160 bash[74864]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11 2024-09-21T01:32:28.807 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:28 smithi160 podman[74883]: 2024-09-21 01:32:28.457761832 +0000 UTC m=+0.635056081 container remove 2d5f5901e912bb140ec7860e517b34ca0c7c219d07e2a6c3927500122360eb86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T01:32:29.107 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:28 smithi160 podman[74968]: 2024-09-21 01:32:28.710433591 +0000 UTC m=+0.015964465 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:32:29.140 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:28 smithi160 podman[74968]: 2024-09-21 01:32:28.808302828 +0000 UTC m=+0.113833694 container create 9e7202a9211724df4faac6f439fabafdf8ff27c785ac75663a5612a1b691c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T01:32:29.140 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:29 smithi160 podman[74968]: 2024-09-21 01:32:29.044040022 +0000 UTC m=+0.349570891 container init 9e7202a9211724df4faac6f439fabafdf8ff27c785ac75663a5612a1b691c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:32:29.141 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:29 smithi160 podman[74968]: 2024-09-21 01:32:29.047261386 +0000 UTC m=+0.352792260 container start 9e7202a9211724df4faac6f439fabafdf8ff27c785ac75663a5612a1b691c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T01:32:29.505 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:29 smithi160 podman[74968]: 2024-09-21 01:32:29.127169692 +0000 UTC m=+0.432700566 container attach 9e7202a9211724df4faac6f439fabafdf8ff27c785ac75663a5612a1b691c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T01:32:29.505 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:29 smithi160 podman[74968]: 2024-09-21 01:32:29.241949978 +0000 UTC m=+0.547480844 container died 9e7202a9211724df4faac6f439fabafdf8ff27c785ac75663a5612a1b691c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T01:32:29.761 INFO:journalctl@ceph.osd.11.smithi160.stdout:Sep 21 01:32:29 smithi160 podman[74991]: 2024-09-21 01:32:29.747438703 +0000 UTC m=+0.499540515 container remove 9e7202a9211724df4faac6f439fabafdf8ff27c785ac75663a5612a1b691c389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T01:32:29.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@osd.11.service' 2024-09-21T01:32:29.804 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:29.804 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-21T01:32:29.804 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-21T01:32:29.804 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.a 2024-09-21T01:32:30.153 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:29 smithi039 systemd[1]: Stopping Ceph mds.a for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:30.653 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a[72338]: 2024-09-21T01:32:30.339+0000 7f7329a00640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:30.653 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:30 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a[72338]: 2024-09-21T01:32:30.339+0000 7f7329a00640 -1 mds.a *** got signal Terminated *** 2024-09-21T01:32:34.638 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:34 smithi039 podman[108679]: 2024-09-21 01:32:34.368847077 +0000 UTC m=+4.262632419 container died 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T01:32:34.903 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:34 smithi039 podman[108679]: 2024-09-21 01:32:34.768933306 +0000 UTC m=+4.662718648 container cleanup 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:32:34.903 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:34 smithi039 bash[108679]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a 2024-09-21T01:32:35.203 INFO:journalctl@ceph.mds.a.smithi039.stdout:Sep 21 01:32:34 smithi039 podman[108691]: 2024-09-21 01:32:34.945740312 +0000 UTC m=+0.566947413 container remove 48088db5c710ce093995e4d7e1f90caac9df6f96d299e2b5206a3dbc10bc6f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:32:35.227 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.a.service' 2024-09-21T01:32:35.298 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:35.298 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-21T01:32:35.298 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-21T01:32:35.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.d 2024-09-21T01:32:35.653 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:35 smithi039 systemd[1]: Stopping Ceph mds.d for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:36.153 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-d[71296]: 2024-09-21T01:32:35.700+0000 7fead4800640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:36.153 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:35 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-d[71296]: 2024-09-21T01:32:35.700+0000 7fead4800640 -1 mds.d *** got signal Terminated *** 2024-09-21T01:32:38.653 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:38 smithi039 podman[108798]: 2024-09-21 01:32:38.377895289 +0000 UTC m=+2.807825724 container died 7e6a80933bc923b14e6f7b437f892df2b419086981d32f4ed908914e44ba9231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:32:39.153 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:38 smithi039 podman[108798]: 2024-09-21 01:32:38.956607233 +0000 UTC m=+3.386537668 container cleanup 7e6a80933bc923b14e6f7b437f892df2b419086981d32f4ed908914e44ba9231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:32:39.153 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:38 smithi039 bash[108798]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-d 2024-09-21T01:32:39.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.d.service' 2024-09-21T01:32:39.495 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:39 smithi039 podman[108822]: 2024-09-21 01:32:39.241352985 +0000 UTC m=+0.855353082 container remove 7e6a80933bc923b14e6f7b437f892df2b419086981d32f4ed908914e44ba9231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-d, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:32:39.495 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:39 smithi039 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.d.service: Deactivated successfully. 2024-09-21T01:32:39.495 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:39 smithi039 systemd[1]: Stopped Ceph mds.d for 61abfb08-77b3-11ef-baf5-efdc52797490. 2024-09-21T01:32:39.495 INFO:journalctl@ceph.mds.d.smithi039.stdout:Sep 21 01:32:39 smithi039 systemd[1]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.d.service: Consumed 1.899s CPU time. 2024-09-21T01:32:39.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:39.520 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-21T01:32:39.520 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-21T01:32:39.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.g 2024-09-21T01:32:39.903 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:39 smithi039 systemd[1]: Stopping Ceph mds.g for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:40.403 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g[72024]: 2024-09-21T01:32:39.963+0000 7fec05600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:40.403 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:39 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g[72024]: 2024-09-21T01:32:39.963+0000 7fec05600640 -1 mds.g *** got signal Terminated *** 2024-09-21T01:32:44.111 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:43 smithi039 podman[108928]: 2024-09-21 01:32:43.723692584 +0000 UTC m=+3.880192868 container died 2b422ce1ca6abab4c85998a6ebd04093c3811da6af74c101e3ce6e90c37f0210 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T01:32:44.387 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:44 smithi039 podman[108928]: 2024-09-21 01:32:44.21863459 +0000 UTC m=+4.375134879 container cleanup 2b422ce1ca6abab4c85998a6ebd04093c3811da6af74c101e3ce6e90c37f0210 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T01:32:44.387 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:44 smithi039 bash[108928]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g 2024-09-21T01:32:44.639 INFO:journalctl@ceph.mds.g.smithi039.stdout:Sep 21 01:32:44 smithi039 podman[108940]: 2024-09-21 01:32:44.387023528 +0000 UTC m=+0.652287666 container remove 2b422ce1ca6abab4c85998a6ebd04093c3811da6af74c101e3ce6e90c37f0210 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T01:32:44.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.g.service' 2024-09-21T01:32:44.740 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:44.740 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-21T01:32:44.740 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-21T01:32:44.741 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.j 2024-09-21T01:32:45.126 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:32:44 smithi039 systemd[1]: Stopping Ceph mds.j for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:45.403 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:32:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j[70914]: 2024-09-21T01:32:45.126+0000 7fd5a1400640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:45.403 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:32:45 smithi039 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j[70914]: 2024-09-21T01:32:45.126+0000 7fd5a1400640 -1 mds.j *** got signal Terminated *** 2024-09-21T01:32:47.085 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:32:46 smithi039 podman[109049]: 2024-09-21 01:32:46.706646261 +0000 UTC m=+1.684540304 container died 5c49624a179c29775823354c27798b6b61afdc2e0054d1c54740e1851345e5f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T01:32:47.403 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:32:47 smithi039 podman[109049]: 2024-09-21 01:32:47.197467918 +0000 UTC m=+2.175361958 container cleanup 5c49624a179c29775823354c27798b6b61afdc2e0054d1c54740e1851345e5f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:32:47.403 INFO:journalctl@ceph.mds.j.smithi039.stdout:Sep 21 01:32:47 smithi039 bash[109049]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-j 2024-09-21T01:32:47.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.j.service' 2024-09-21T01:32:47.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:47.671 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-21T01:32:47.671 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-21T01:32:47.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.b 2024-09-21T01:32:47.964 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:47 smithi150 systemd[1]: Stopping Ceph mds.b for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:48.221 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: 2024-09-21T01:32:47.963+0000 7f526ca0a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:48.222 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:47 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b[64452]: 2024-09-21T01:32:47.963+0000 7f526ca0a640 -1 mds.b *** got signal Terminated *** 2024-09-21T01:32:51.414 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:51 smithi150 podman[73858]: 2024-09-21 01:32:51.109355181 +0000 UTC m=+3.248442102 container died 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2) 2024-09-21T01:32:51.414 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:51 smithi150 podman[73858]: 2024-09-21 01:32:51.413683796 +0000 UTC m=+3.552770718 container cleanup 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T01:32:51.782 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:51 smithi150 bash[73858]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b 2024-09-21T01:32:51.783 INFO:journalctl@ceph.mds.b.smithi150.stdout:Sep 21 01:32:51 smithi150 podman[73882]: 2024-09-21 01:32:51.656414157 +0000 UTC m=+0.539318734 container remove 559a4fc2128e120398aca523a0924898f0e533bf158c154240e0270c88a7a964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T01:32:51.799 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.b.service' 2024-09-21T01:32:51.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:51.835 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-21T01:32:51.835 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-21T01:32:51.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.e 2024-09-21T01:32:52.064 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:32:51 smithi150 systemd[1]: Stopping Ceph mds.e for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:52.563 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:32:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e[63895]: 2024-09-21T01:32:52.208+0000 7f8f21d98640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:52.564 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:32:52 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e[63895]: 2024-09-21T01:32:52.208+0000 7f8f21d98640 -1 mds.e *** got signal Terminated *** 2024-09-21T01:32:56.509 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:32:56 smithi150 podman[73992]: 2024-09-21 01:32:56.221372862 +0000 UTC m=+4.169134241 container died 11c0020994b70a053eb21bb8d698b46b6bf8be9adf423d3629d5948aaf3c0dcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T01:32:56.814 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:32:56 smithi150 podman[73992]: 2024-09-21 01:32:56.608819007 +0000 UTC m=+4.556580385 container cleanup 11c0020994b70a053eb21bb8d698b46b6bf8be9adf423d3629d5948aaf3c0dcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T01:32:56.814 INFO:journalctl@ceph.mds.e.smithi150.stdout:Sep 21 01:32:56 smithi150 bash[73992]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-e 2024-09-21T01:32:57.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.e.service' 2024-09-21T01:32:57.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:57.037 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-21T01:32:57.037 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-21T01:32:57.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.h 2024-09-21T01:32:57.564 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:32:57 smithi150 systemd[1]: Stopping Ceph mds.h for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:32:57.564 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:32:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h[64165]: 2024-09-21T01:32:57.361+0000 7f989f696640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:32:57.564 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:32:57 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h[64165]: 2024-09-21T01:32:57.361+0000 7f989f696640 -1 mds.h *** got signal Terminated *** 2024-09-21T01:32:59.380 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:32:59 smithi150 podman[74111]: 2024-09-21 01:32:59.108728904 +0000 UTC m=+1.856101354 container died 2e6bdd1c228035ba75de9dad8d2bf787373de26abb5fa5b8c8c8781cec1268e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T01:32:59.739 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:32:59 smithi150 podman[74111]: 2024-09-21 01:32:59.471142666 +0000 UTC m=+2.218515125 container cleanup 2e6bdd1c228035ba75de9dad8d2bf787373de26abb5fa5b8c8c8781cec1268e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2) 2024-09-21T01:32:59.739 INFO:journalctl@ceph.mds.h.smithi150.stdout:Sep 21 01:32:59 smithi150 bash[74111]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-h 2024-09-21T01:32:59.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.h.service' 2024-09-21T01:32:59.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:32:59.925 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-21T01:32:59.925 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-21T01:32:59.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.k 2024-09-21T01:33:00.314 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:00 smithi150 systemd[1]: Stopping Ceph mds.k for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:33:00.314 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k[63611]: 2024-09-21T01:33:00.241+0000 7fb49c6b8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:33:00.314 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:00 smithi150 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k[63611]: 2024-09-21T01:33:00.241+0000 7fb49c6b8640 -1 mds.k *** got signal Terminated *** 2024-09-21T01:33:03.241 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:02 smithi150 podman[74234]: 2024-09-21 01:33:02.925360093 +0000 UTC m=+2.774847334 container died b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:33:03.241 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:03 smithi150 podman[74234]: 2024-09-21 01:33:03.241453608 +0000 UTC m=+3.090940845 container cleanup b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T01:33:03.564 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:03 smithi150 bash[74234]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k 2024-09-21T01:33:03.564 INFO:journalctl@ceph.mds.k.smithi150.stdout:Sep 21 01:33:03 smithi150 podman[74248]: 2024-09-21 01:33:03.475541476 +0000 UTC m=+0.534903807 container remove b1f17fc132547dd87f84f240a9adf727fcde811c0c8ba0c5201401fcd59e93bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-k, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T01:33:03.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.k.service' 2024-09-21T01:33:03.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:33:03.642 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-21T01:33:03.643 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-21T01:33:03.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.c 2024-09-21T01:33:03.931 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:03 smithi160 systemd[1]: Stopping Ceph mds.c for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:33:04.356 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c[64241]: 2024-09-21T01:33:03.929+0000 7f218a031640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:33:04.356 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:03 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c[64241]: 2024-09-21T01:33:03.929+0000 7f218a031640 -1 mds.c *** got signal Terminated *** 2024-09-21T01:33:07.356 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:07 smithi160 podman[75084]: 2024-09-21 01:33:07.093173929 +0000 UTC m=+3.261463070 container died a5b3475708b0a8400cc5ae90205732b16c0388a64cb120dac2397cd882c182c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True) 2024-09-21T01:33:07.857 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:07 smithi160 podman[75084]: 2024-09-21 01:33:07.446284976 +0000 UTC m=+3.614574077 container cleanup a5b3475708b0a8400cc5ae90205732b16c0388a64cb120dac2397cd882c182c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T01:33:07.857 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:07 smithi160 bash[75084]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c 2024-09-21T01:33:07.857 INFO:journalctl@ceph.mds.c.smithi160.stdout:Sep 21 01:33:07 smithi160 podman[75097]: 2024-09-21 01:33:07.622460087 +0000 UTC m=+0.505685232 container remove a5b3475708b0a8400cc5ae90205732b16c0388a64cb120dac2397cd882c182c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-c, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:33:07.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.c.service' 2024-09-21T01:33:07.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:33:07.914 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-21T01:33:07.914 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-21T01:33:07.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.f 2024-09-21T01:33:08.225 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:33:07 smithi160 systemd[1]: Stopping Ceph mds.f for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:33:08.606 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:33:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f[63967]: 2024-09-21T01:33:08.224+0000 7f7f097ca640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:33:08.606 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:33:08 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f[63967]: 2024-09-21T01:33:08.224+0000 7f7f097ca640 -1 mds.f *** got signal Terminated *** 2024-09-21T01:33:11.356 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:33:11 smithi160 podman[75206]: 2024-09-21 01:33:11.099241779 +0000 UTC m=+2.965885974 container died a0e492f5e6194f7c5b7c8c8771ef501cb523742d2ba6f3bc9bc829e99ebac444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:33:11.659 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:33:11 smithi160 podman[75206]: 2024-09-21 01:33:11.399932525 +0000 UTC m=+3.266576720 container cleanup a0e492f5e6194f7c5b7c8c8771ef501cb523742d2ba6f3bc9bc829e99ebac444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T01:33:11.659 INFO:journalctl@ceph.mds.f.smithi160.stdout:Sep 21 01:33:11 smithi160 bash[75206]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-f 2024-09-21T01:33:11.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.f.service' 2024-09-21T01:33:11.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:33:11.863 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-21T01:33:11.863 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-21T01:33:11.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.i 2024-09-21T01:33:12.358 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:11 smithi160 systemd[1]: Stopping Ceph mds.i for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:33:12.358 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i[64525]: 2024-09-21T01:33:12.170+0000 7f8a6455c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:33:12.358 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:12 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i[64525]: 2024-09-21T01:33:12.170+0000 7f8a6455c640 -1 mds.i *** got signal Terminated *** 2024-09-21T01:33:15.357 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:15 smithi160 podman[75328]: 2024-09-21 01:33:15.098257516 +0000 UTC m=+3.020679472 container died f5da40691c52525672cce524b28f074b72fc2c54047116d1b9e1ef175ce0b1c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:33:15.844 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:15 smithi160 podman[75328]: 2024-09-21 01:33:15.436955036 +0000 UTC m=+3.359376992 container cleanup f5da40691c52525672cce524b28f074b72fc2c54047116d1b9e1ef175ce0b1c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T01:33:15.844 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:15 smithi160 bash[75328]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i 2024-09-21T01:33:15.845 INFO:journalctl@ceph.mds.i.smithi160.stdout:Sep 21 01:33:15 smithi160 podman[75343]: 2024-09-21 01:33:15.604884903 +0000 UTC m=+0.506041510 container remove f5da40691c52525672cce524b28f074b72fc2c54047116d1b9e1ef175ce0b1c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T01:33:15.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.i.service' 2024-09-21T01:33:15.897 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:33:15.897 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-21T01:33:15.897 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-21T01:33:15.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.l 2024-09-21T01:33:16.215 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 01:33:15 smithi160 systemd[1]: Stopping Ceph mds.l for 61abfb08-77b3-11ef-baf5-efdc52797490... 2024-09-21T01:33:16.606 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 01:33:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l[63698]: 2024-09-21T01:33:16.215+0000 7f16dd2f7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:33:16.607 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 01:33:16 smithi160 ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l[63698]: 2024-09-21T01:33:16.215+0000 7f16dd2f7640 -1 mds.l *** got signal Terminated *** 2024-09-21T01:33:20.563 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 01:33:20 smithi160 podman[75450]: 2024-09-21 01:33:20.221055016 +0000 UTC m=+4.103560017 container died c202ad70b51127b9ee720c7ed9ffdf1522b437ebcfb8fc3495a214f8d71c9d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T01:33:20.857 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 01:33:20 smithi160 podman[75450]: 2024-09-21 01:33:20.665711805 +0000 UTC m=+4.548216797 container cleanup c202ad70b51127b9ee720c7ed9ffdf1522b437ebcfb8fc3495a214f8d71c9d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T01:33:20.857 INFO:journalctl@ceph.mds.l.smithi160.stdout:Sep 21 01:33:20 smithi160 bash[75450]: ceph-61abfb08-77b3-11ef-baf5-efdc52797490-mds-l 2024-09-21T01:33:21.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61abfb08-77b3-11ef-baf5-efdc52797490@mds.l.service' 2024-09-21T01:33:21.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:33:21.113 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-21T01:33:21.113 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T01:33:21.398 INFO:teuthology.orchestra.run.smithi039.stdout:Deleting cluster with fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:33:25.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T01:33:25.557 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:33:28.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T01:33:28.374 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:33:30.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:33:30.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:33:30.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:33:30.953 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T01:33:30.955 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi039/crash 2024-09-21T01:33:30.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash -- . 2024-09-21T01:33:31.024 INFO:teuthology.orchestra.run.smithi039.stderr:tar: /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T01:33:31.024 INFO:teuthology.orchestra.run.smithi039.stderr:tar: Error is not recoverable: exiting now 2024-09-21T01:33:31.027 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi150/crash 2024-09-21T01:33:31.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash -- . 2024-09-21T01:33:31.064 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T01:33:31.064 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-09-21T01:33:31.066 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi160/crash 2024-09-21T01:33:31.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash -- . 2024-09-21T01:33:31.123 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T01:33:31.123 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2024-09-21T01:33:31.124 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T01:33:31.124 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-21T01:33:31.292 INFO:tasks.cephadm:Compressing logs... 2024-09-21T01:33:31.293 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T01:33:31.297 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T01:33:31.300 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T01:33:31.325 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T01:33:31.325 INFO:teuthology.orchestra.run.smithi160.stderr:: No such file or directory 2024-09-21T01:33:31.325 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T01:33:31.325 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.i.log 2024-09-21T01:33:31.325 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-21T01:33:31.326 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log: /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T01:33:31.326 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.11.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.l.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.8.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.z.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.c.log 2024-09-21T01:33:31.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.f.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.10.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:: No such file or directory 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.b.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.k.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T01:33:31.332 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-21T01:33:31.333 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log 2024-09-21T01:33:31.333 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.h.log 2024-09-21T01:33:31.333 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-21T01:33:31.333 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.e.log 2024-09-21T01:33:31.333 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T01:33:31.371 INFO:teuthology.orchestra.run.smithi039.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T01:33:31.371 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-21T01:33:31.372 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-21T01:33:31.372 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.9.log 2024-09-21T01:33:31.373 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T01:33:31.374 INFO:teuthology.orchestra.run.smithi039.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-21T01:33:31.375 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T01:33:31.375 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.a.log 2024-09-21T01:33:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T01:33:31.377 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-21T01:33:31.378 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.g.log 2024-09-21T01:33:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-21T01:33:31.379 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log 2024-09-21T01:33:31.380 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.d.log 2024-09-21T01:33:31.381 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-21T01:33:31.382 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.j.log 2024-09-21T01:33:31.383 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-21T01:33:31.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T01:33:31.385 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.j.log: /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.3.log: 87.1% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T01:33:31.390 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/cephadm.log: 95.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.g.log.gz 2024-09-21T01:33:31.393 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.f.log: /var/log/ceph/cephadm.log: 90.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T01:33:31.395 INFO:teuthology.orchestra.run.smithi039.stderr: 94.7% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.d.log.gz 2024-09-21T01:33:31.397 INFO:teuthology.orchestra.run.smithi039.stderr: 95.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.j.log.gz 2024-09-21T01:33:31.412 INFO:teuthology.orchestra.run.smithi160.stderr: 82.3% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T01:33:31.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.e.log: /var/log/ceph/cephadm.log: 90.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T01:33:31.495 INFO:teuthology.orchestra.run.smithi150.stderr: 93.3% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T01:33:31.513 INFO:teuthology.orchestra.run.smithi160.stderr: 91.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T01:33:31.531 INFO:teuthology.orchestra.run.smithi039.stderr: 93.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T01:33:31.536 INFO:teuthology.orchestra.run.smithi150.stderr: 94.9% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.e.log.gz 2024-09-21T01:33:31.537 INFO:teuthology.orchestra.run.smithi039.stderr: 84.6% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T01:33:31.555 INFO:teuthology.orchestra.run.smithi150.stderr: 82.3% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T01:33:31.559 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.z.log.gz 2024-09-21T01:33:31.601 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T01:33:31.615 INFO:teuthology.orchestra.run.smithi150.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T01:33:31.633 INFO:teuthology.orchestra.run.smithi039.stderr: 91.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T01:33:31.767 INFO:teuthology.orchestra.run.smithi150.stderr: 94.9% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.k.log.gz 2024-09-21T01:33:31.844 INFO:teuthology.orchestra.run.smithi039.stderr: 90.1% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T01:33:32.537 INFO:teuthology.orchestra.run.smithi160.stderr: 87.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T01:33:32.541 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T01:33:32.607 INFO:teuthology.orchestra.run.smithi150.stderr: 91.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T01:33:32.933 INFO:teuthology.orchestra.run.smithi039.stderr: 90.5% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T01:33:33.046 INFO:teuthology.orchestra.run.smithi160.stderr: 94.9% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.l.log.gz 2024-09-21T01:33:34.861 INFO:teuthology.orchestra.run.smithi160.stderr: 91.8% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-21T01:33:38.087 INFO:teuthology.orchestra.run.smithi039.stderr: 90.3% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-21T01:34:16.791 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-21T01:34:57.675 INFO:teuthology.orchestra.run.smithi150.stderr: 94.9% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.h.log.gz 2024-09-21T01:36:12.945 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.10.log.gz 2024-09-21T01:36:35.162 INFO:teuthology.orchestra.run.smithi039.stderr: 91.4% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-21T01:37:03.975 INFO:teuthology.orchestra.run.smithi039.stderr: 91.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-21T01:37:04.971 INFO:teuthology.orchestra.run.smithi160.stderr: 91.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.8.log.gz 2024-09-21T01:37:45.057 INFO:teuthology.orchestra.run.smithi160.stderr: 94.9% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.f.log.gz 2024-09-21T01:37:53.993 INFO:teuthology.orchestra.run.smithi160.stderr: 95.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.c.log.gz 2024-09-21T01:38:00.257 INFO:teuthology.orchestra.run.smithi039.stderr: 91.1% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.9.log.gz 2024-09-21T01:38:14.877 INFO:teuthology.orchestra.run.smithi039.stderr: 91.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-21T01:39:27.832 INFO:teuthology.orchestra.run.smithi160.stderr: 91.4% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.11.log.gz 2024-09-21T01:40:11.550 INFO:teuthology.orchestra.run.smithi160.stderr: 91.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-21T01:40:15.330 INFO:teuthology.orchestra.run.smithi039.stderr: 92.8% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.a.log.gz 2024-09-21T01:40:15.333 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T01:40:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:real 6m44.007s 2024-09-21T01:40:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:user 5m37.990s 2024-09-21T01:40:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m35.148s 2024-09-21T01:40:36.158 INFO:teuthology.orchestra.run.smithi150.stderr: 91.5% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-21T01:42:00.080 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-21T01:42:26.587 INFO:teuthology.orchestra.run.smithi150.stderr: 91.2% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-21T01:42:35.846 INFO:teuthology.orchestra.run.smithi160.stderr: 91.1% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-21T01:44:25.204 INFO:teuthology.orchestra.run.smithi160.stderr: 92.8% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.i.log.gz 2024-09-21T01:44:25.206 INFO:teuthology.orchestra.run.smithi160.stderr: 2024-09-21T01:44:25.206 INFO:teuthology.orchestra.run.smithi160.stderr:real 10m53.891s 2024-09-21T01:44:25.206 INFO:teuthology.orchestra.run.smithi160.stderr:user 5m59.429s 2024-09-21T01:44:25.206 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m15.007s 2024-09-21T01:45:28.901 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/61abfb08-77b3-11ef-baf5-efdc52797490/ceph-mds.b.log.gz 2024-09-21T01:45:29.254 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-21T01:45:29.255 INFO:teuthology.orchestra.run.smithi150.stderr:real 11m57.939s 2024-09-21T01:45:29.255 INFO:teuthology.orchestra.run.smithi150.stderr:user 6m16.096s 2024-09-21T01:45:29.255 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m15.776s 2024-09-21T01:45:29.255 INFO:tasks.cephadm:Archiving logs... 2024-09-21T01:45:29.256 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi039/log 2024-09-21T01:45:29.257 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T01:46:19.746 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi150/log 2024-09-21T01:46:19.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T01:47:18.516 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi160/log 2024-09-21T01:47:18.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T01:48:14.563 INFO:tasks.cephadm:Removing cluster... 2024-09-21T01:48:14.563 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --force 2024-09-21T01:48:14.868 INFO:teuthology.orchestra.run.smithi039.stdout:Deleting cluster with fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:48:16.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --force 2024-09-21T01:48:17.147 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:48:17.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61abfb08-77b3-11ef-baf5-efdc52797490 --force 2024-09-21T01:48:18.082 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 61abfb08-77b3-11ef-baf5-efdc52797490 2024-09-21T01:48:18.816 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T01:48:18.816 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T01:48:18.857 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T01:48:18.875 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T01:48:18.893 INFO:tasks.cephadm:Teardown complete 2024-09-21T01:48:18.893 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T01:48:18.904 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T01:48:18.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T01:48:18.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T01:48:18.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T01:48:19.064 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T01:48:19.087 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T01:48:19.088 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:48:19.092 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:48:19.096 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:48:19.109 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:48:19.110 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:48:19.123 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:48:19.131 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:48:19.131 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-09-21T01:48:19.131 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +301us[ +303us] +/- 48ms 2024-09-21T01:48:19.131 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -903us[ -903us] +/- 58ms 2024-09-21T01:48:19.131 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +671us[ +671us] +/- 68ms 2024-09-21T01:48:19.131 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:48:19.132 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:48:19.132 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-21T01:48:19.132 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +284us[ +309us] +/- 48ms 2024-09-21T01:48:19.133 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 -963us[ -939us] +/- 58ms 2024-09-21T01:48:19.133 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +663us[ +687us] +/- 68ms 2024-09-21T01:48:19.133 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:48:19.215 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:48:19.215 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-09-21T01:48:19.215 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv01.front.sepia.ceph.com 3 7 377 6 +329us[ +409us] +/- 48ms 2024-09-21T01:48:19.216 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 -938us[ -938us] +/- 58ms 2024-09-21T01:48:19.216 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +689us[ +689us] +/- 68ms 2024-09-21T01:48:19.216 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:48:19.217 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T01:48:19.228 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T01:48:19.228 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T01:48:19.271 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T01:48:19.458 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T01:48:19.540 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T01:48:19.646 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T01:48:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:48:19.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T01:48:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:48:19.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T01:48:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:48:19.767 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T01:48:19.778 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T01:48:19.823 INFO:teuthology.task.internal:Duration was 3917.924132 seconds 2024-09-21T01:48:19.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T01:48:19.856 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T01:48:19.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T01:48:19.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T01:48:19.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T01:48:19.926 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:48:19.930 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:48:19.961 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:48:20.405 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T01:48:20.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T01:48:20.406 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T01:48:20.457 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-09-21T01:48:20.457 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T01:48:20.486 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T01:48:20.486 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T01:48:20.525 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T01:48:20.526 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T01:48:20.530 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T01:48:20.532 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T01:48:20.592 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T01:48:20.592 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T01:48:20.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T01:48:20.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T01:48:21.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T01:48:21.898 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T01:48:21.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T01:48:21.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T01:48:22.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T01:48:22.053 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T01:48:22.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T01:48:22.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T01:48:22.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T01:48:22.119 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-09-21T01:48:22.138 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2024-09-21T01:48:22.153 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-21T01:48:22.183 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T01:48:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:48:22.220 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T01:48:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:48:22.238 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T01:48:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:48:22.256 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T01:48:22.323 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T01:48:22.324 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi039 2024-09-21T01:48:22.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T01:48:22.429 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi150 2024-09-21T01:48:22.429 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T01:48:22.473 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913344/remote/smithi160 2024-09-21T01:48:22.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T01:48:22.518 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T01:48:22.518 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T01:48:22.523 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T01:48:22.526 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T01:48:22.573 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T01:48:22.587 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T01:48:22.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T01:48:22.624 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T01:48:22.624 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T01:48:22.630 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T01:48:22.633 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T01:48:22.648 INFO:teuthology.orchestra.run.smithi150.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 01:48 /home/ubuntu/cephtest 2024-09-21T01:48:22.650 INFO:teuthology.orchestra.run.smithi160.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 01:48 /home/ubuntu/cephtest 2024-09-21T01:48:22.660 INFO:teuthology.orchestra.run.smithi039.stdout: 393233 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 01:48 /home/ubuntu/cephtest 2024-09-21T01:48:22.662 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T01:48:22.674 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T01:48:22.735 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 3917.924132347107 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-21T01:48:22.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T01:48:22.820 INFO:teuthology.run:pass